* [PATCH 1/2] ui-snapshot: add support for zstd compression @ 2020-02-26 8:21 list 2020-02-26 8:21 ` [PATCH 2/2] tests: add tests for xz compressed snapshots list ` (2 more replies) 0 siblings, 3 replies; 8+ messages in thread From: list @ 2020-02-26 8:21 UTC (permalink / raw) From: Christian Hesse <mail at eworm.de> This patch adds support for zstd [0] compressed snapshots (*.tar.zst). [0] https://www.zstd.net/ Signed-off-by: Christian Hesse <mail at eworm.de> --- cgitrc.5.txt | 5 +++-- tests/setup.sh | 2 +- tests/t0107-snapshot.sh | 42 +++++++++++++++++++++++++++++++++++++++++ ui-snapshot.c | 7 +++++++ 4 files changed, 53 insertions(+), 3 deletions(-) diff --git a/cgitrc.5.txt b/cgitrc.5.txt index 4ad3e64..b0ec698 100644 --- a/cgitrc.5.txt +++ b/cgitrc.5.txt @@ -407,8 +407,9 @@ side-by-side-diffs:: snapshots:: Text which specifies the default set of snapshot formats that cgit generates links for. The value is a space-separated list of zero or - more of the values "tar", "tar.gz", "tar.bz2", "tar.xz", "tar.lz" and - "zip". The special value "all" enables all snapshot formats. + more of the values "tar", "tar.gz", "tar.bz2", "tar.lz", "tar.xz", + "tar.zst" and "zip". The special value "all" enables all snapshot + formats. Default value: none. source-filter:: diff --git a/tests/setup.sh b/tests/setup.sh index 69e47e6..e09f7c5 100755 --- a/tests/setup.sh +++ b/tests/setup.sh @@ -104,7 +104,7 @@ virtual-root=/ cache-root=$PWD/cache cache-size=1021 -snapshots=tar.gz tar.bz tar.lz zip +snapshots=tar.gz tar.bz tar.lz tar.zst zip enable-log-filecount=1 enable-log-linecount=1 summary-log=5 diff --git a/tests/t0107-snapshot.sh b/tests/t0107-snapshot.sh index a845ad9..95ad624 100755 --- a/tests/t0107-snapshot.sh +++ b/tests/t0107-snapshot.sh @@ -80,6 +80,48 @@ test_expect_success LZIP 'verify untarred file-5' ' test_line_count = 1 master/file-5 ' +if test -n "$(which zstd 2>/dev/null)"; then + test_set_prereq ZSTD +else + say 'Skipping ZSTD validation tests: zstd not found' +fi + +test_expect_success ZSTD 'get foo/snapshot/master.tar.zst' ' + cgit_url "foo/snapshot/master.tar.zst" >tmp +' + +test_expect_success ZSTD 'check html headers' ' + head -n 1 tmp | + grep "Content-Type: application/x-zstd" && + + head -n 2 tmp | + grep "Content-Disposition: inline; filename=.master.tar.zst." +' + +test_expect_success ZSTD 'strip off the header lines' ' + strip_headers <tmp >master.tar.zst +' + +test_expect_success ZSTD 'verify zstd format' ' + zstd --test master.tar.zst && + cp master.tar.zst /tmp/. +' + +test_expect_success ZSTD 'untar' ' + rm -rf master && + tar --zstd -xf master.tar.zst +' + +test_expect_success ZSTD 'count files' ' + ls master/ >output && + test_line_count = 5 output +' + +test_expect_success ZSTD 'verify untarred file-5' ' + grep "^5$" master/file-5 && + test_line_count = 1 master/file-5 +' + test_expect_success 'get foo/snapshot/master.zip' ' cgit_url "foo/snapshot/master.zip" >tmp ' diff --git a/ui-snapshot.c b/ui-snapshot.c index 92cde42..db9f51f 100644 --- a/ui-snapshot.c +++ b/ui-snapshot.c @@ -91,6 +91,12 @@ static int write_tar_xz_archive(const char *hex, const char *prefix) return write_compressed_tar_archive(hex, prefix, argv); } +static int write_tar_zstd_archive(const char *hex, const char *prefix) +{ + char *argv[] = { "zstd", NULL }; + return write_compressed_tar_archive(hex, prefix, argv); +} + const struct cgit_snapshot_format cgit_snapshot_formats[] = { /* .tar must remain the 0 index */ { ".tar", "application/x-tar", write_tar_archive }, @@ -98,6 +104,7 @@ const struct cgit_snapshot_format cgit_snapshot_formats[] = { { ".tar.bz2", "application/x-bzip2", write_tar_bzip2_archive }, { ".tar.lz", "application/x-lzip", write_tar_lzip_archive }, { ".tar.xz", "application/x-xz", write_tar_xz_archive }, + { ".tar.zst", "application/x-zstd", write_tar_zstd_archive }, { ".zip", "application/x-zip", write_zip_archive }, { NULL } }; ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 2/2] tests: add tests for xz compressed snapshots 2020-02-26 8:21 [PATCH 1/2] ui-snapshot: add support for zstd compression list @ 2020-02-26 8:21 ` list 2020-02-26 9:19 ` Jason 2020-02-26 9:16 ` [PATCH 1/2] ui-snapshot: add support for zstd compression Jason 2020-02-26 21:24 ` [PATCH v2 " list 2 siblings, 1 reply; 8+ messages in thread From: list @ 2020-02-26 8:21 UTC (permalink / raw) From: Christian Hesse <mail at eworm.de> Signed-off-by: Christian Hesse <mail at eworm.de> --- tests/setup.sh | 2 +- tests/t0107-snapshot.sh | 42 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/tests/setup.sh b/tests/setup.sh index e09f7c5..5879348 100755 --- a/tests/setup.sh +++ b/tests/setup.sh @@ -104,7 +104,7 @@ virtual-root=/ cache-root=$PWD/cache cache-size=1021 -snapshots=tar.gz tar.bz tar.lz tar.zst zip +snapshots=tar.gz tar.bz tar.lz tar.xz tar.zst zip enable-log-filecount=1 enable-log-linecount=1 summary-log=5 diff --git a/tests/t0107-snapshot.sh b/tests/t0107-snapshot.sh index 95ad624..7e343d9 100755 --- a/tests/t0107-snapshot.sh +++ b/tests/t0107-snapshot.sh @@ -122,6 +122,48 @@ test_expect_success ZSTD 'verify untarred file-5' ' test_line_count = 1 master/file-5 ' +if test -n "$(which xz 2>/dev/null)"; then + test_set_prereq XZ +else + say 'Skipping XZ validation tests: xz not found' +fi + +test_expect_success XZ 'get foo/snapshot/master.tar.xz' ' + cgit_url "foo/snapshot/master.tar.xz" >tmp +' + +test_expect_success XZ 'check html headers' ' + head -n 1 tmp | + grep "Content-Type: application/x-xz" && + + head -n 2 tmp | + grep "Content-Disposition: inline; filename=.master.tar.xz." +' + +test_expect_success XZ 'strip off the header lines' ' + strip_headers <tmp >master.tar.xz +' + +test_expect_success XZ 'verify xz format' ' + xz --test master.tar.xz && + cp master.tar.xz /tmp/. +' + +test_expect_success XZ 'untar' ' + rm -rf master && + tar --xz -xf master.tar.xz +' + +test_expect_success XZ 'count files' ' + ls master/ >output && + test_line_count = 5 output +' + +test_expect_success XZ 'verify untarred file-5' ' + grep "^5$" master/file-5 && + test_line_count = 1 master/file-5 +' + test_expect_success 'get foo/snapshot/master.zip' ' cgit_url "foo/snapshot/master.zip" >tmp ' ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 2/2] tests: add tests for xz compressed snapshots 2020-02-26 8:21 ` [PATCH 2/2] tests: add tests for xz compressed snapshots list @ 2020-02-26 9:19 ` Jason 0 siblings, 0 replies; 8+ messages in thread From: Jason @ 2020-02-26 9:19 UTC (permalink / raw) Looks good, queue it up, thanks. ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/2] ui-snapshot: add support for zstd compression 2020-02-26 8:21 [PATCH 1/2] ui-snapshot: add support for zstd compression list 2020-02-26 8:21 ` [PATCH 2/2] tests: add tests for xz compressed snapshots list @ 2020-02-26 9:16 ` Jason 2020-02-26 10:18 ` list [not found] ` <alpine.DEB.2.22.394.2002261715550.3793@Zeta> 2020-02-26 21:24 ` [PATCH v2 " list 2 siblings, 2 replies; 8+ messages in thread From: Jason @ 2020-02-26 9:16 UTC (permalink / raw) Wow, thanks for doing this. I had just been discussing this with Unit193, who submitted a similar but not as complete patch a few days ago to me on IRC. We had been discussing which compression level to pass to zstd or to leave it as default. I think Unit193 had some thoughts on that, so CC'ing him in case he wants to jump in. Barring any change as a result of that, happy to merge this. Thanks. ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/2] ui-snapshot: add support for zstd compression 2020-02-26 9:16 ` [PATCH 1/2] ui-snapshot: add support for zstd compression Jason @ 2020-02-26 10:18 ` list 2020-02-26 10:28 ` list [not found] ` <alpine.DEB.2.22.394.2002261715550.3793@Zeta> 1 sibling, 1 reply; 8+ messages in thread From: list @ 2020-02-26 10:18 UTC (permalink / raw) "Jason A. Donenfeld" <Jason at zx2c4.com> on Wed, 2020/02/26 17:16: > We had been discussing which compression level to pass to zstd or to > leave it as default. I think Unit193 had some thoughts on that, so > CC'ing him in case he wants to jump in. Thought about that myself after I sent the mail... This is a trade-off... Do we prefer faster speed or better compression? We can not have both, just either of both or a mix. And do we want to go with multi-threaded compression? The default compression level is 3. For our Arch packages we decided to go with level 20... Well, the packages are compressed once and decompressed on a magnitude of machines - so it makes sense there. I would propose to take something in between... I think compression level 10 gives reasonable results. Any thoughts? Some numbers: % for I in $(seq 3 19); do zstd -$I < git-2.25.1.tar > git-2.25.1-$I.tar.zst; done level 3: 0,30s user 0,03s system 105% cpu 0,306 total level 4: 0,30s user 0,03s system 107% cpu 0,307 total level 5: 0,57s user 0,03s system 103% cpu 0,574 total level 6: 0,72s user 0,04s system 103% cpu 0,739 total level 7: 0,90s user 0,04s system 102% cpu 0,912 total level 8: 1,13s user 0,03s system 102% cpu 1,134 total level 9: 1,59s user 0,02s system 100% cpu 1,598 total level 10: 1,98s user 0,04s system 100% cpu 2,002 total level 11: 2,28s user 0,05s system 100% cpu 2,318 total level 12: 3,50s user 0,05s system 100% cpu 3,536 total level 13: 4,70s user 0,05s system 100% cpu 4,749 total level 14: 6,52s user 0,06s system 99% cpu 6,603 total level 15: 7,06s user 0,09s system 100% cpu 7,146 total level 16: 9,35s user 0,10s system 99% cpu 9,456 total level 17: 12,00s user 0,08s system 99% cpu 12,118 total level 18: 14,26s user 0,08s system 99% cpu 14,390 total level 19: 19,66s user 0,10s system 99% cpu 19,835 total % for I in $(seq 3 19); do zstd -T0 -$I < git-2.25.1.tar > git-2.25.1-$I-T0.tar.zst; done level 3: 0,41s user 0,05s system 182% cpu 0,251 total level 4: 0,52s user 0,05s system 200% cpu 0,285 total level 5: 0,81s user 0,04s system 195% cpu 0,434 total level 6: 1,59s user 0,06s system 188% cpu 0,877 total level 7: 1,25s user 0,04s system 189% cpu 0,681 total level 8: 1,63s user 0,04s system 187% cpu 0,888 total level 9: 2,12s user 0,05s system 187% cpu 1,156 total level 10: 2,58s user 0,07s system 178% cpu 1,482 total level 11: 2,85s user 0,07s system 176% cpu 1,650 total level 12: 4,33s user 0,12s system 173% cpu 2,560 total level 13: 5,50s user 0,07s system 188% cpu 2,948 total level 14: 6,42s user 0,09s system 192% cpu 3,389 total level 15: 8,21s user 0,13s system 185% cpu 4,492 total level 16: 10,28s user 0,07s system 179% cpu 5,778 total level 17: 12,08s user 0,10s system 117% cpu 10,411 total level 18: 16,24s user 0,11s system 114% cpu 14,271 total level 19: 24,78s user 0,21s system 125% cpu 19,843 total % xz < git-2.25.1.tar > git-2.25.1.tar.xz 17,07s user 0,26s system 139% cpu 12,453 total % time gzip < git-2.25.1.tar > git-2.25.1.tar.gz 2,16s user 0,04s system 361% cpu 0,607 total 37M git-2.25.1.tar 8,7M git-2.25.1.tar.gz 5,9M git-2.25.1.tar.xz 8,2M git-2.25.1-3.tar.zst 8,1M git-2.25.1-4.tar.zst 7,9M git-2.25.1-5.tar.zst 7,6M git-2.25.1-6.tar.zst 7,3M git-2.25.1-7.tar.zst 7,1M git-2.25.1-8.tar.zst 7,0M git-2.25.1-9.tar.zst 6,8M git-2.25.1-10.tar.zst 6,8M git-2.25.1-11.tar.zst 6,7M git-2.25.1-12.tar.zst 6,5M git-2.25.1-13.tar.zst 6,4M git-2.25.1-14.tar.zst 6,4M git-2.25.1-15.tar.zst 6,1M git-2.25.1-16.tar.zst 6,0M git-2.25.1-17.tar.zst 5,9M git-2.25.1-18.tar.zst 5,9M git-2.25.1-19.tar.zst -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);} -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20200226/3ad5fb8b/attachment.asc> ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/2] ui-snapshot: add support for zstd compression 2020-02-26 10:18 ` list @ 2020-02-26 10:28 ` list 0 siblings, 0 replies; 8+ messages in thread From: list @ 2020-02-26 10:28 UTC (permalink / raw) Christian Hesse <list at eworm.de> on Wed, 2020/02/26 11:18: > % time gzip < git-2.25.1.tar > git-2.25.1.tar.gz > 2,16s user 0,04s system 361% cpu 0,607 total This was multi-threaded pigz... Plain gzip looks like this: % /usr/bin/gzip < git-2.25.1.tar > git-2.25.1.tar.gz 1,39s user 0,01s system 98% cpu 1,425 total -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);} -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20200226/b4467d49/attachment-0001.asc> ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <alpine.DEB.2.22.394.2002261715550.3793@Zeta>]
* [PATCH 1/2] ui-snapshot: add support for zstd compression [not found] ` <alpine.DEB.2.22.394.2002261715550.3793@Zeta> @ 2020-02-26 22:38 ` list 0 siblings, 0 replies; 8+ messages in thread From: list @ 2020-02-26 22:38 UTC (permalink / raw) Unit 193 <unit193 at ubuntu.com> on Wed, 2020/02/26 17:21: > > Wow, thanks for doing this. I had just been discussing this with > > Unit193, who submitted a similar but not as complete patch a few days > > ago to me on IRC. > > Indeed! Thanks for looking into this! I just implemented this after the lzip patch was merged... Did not even know you discussed this. :-p > > We had been discussing which compression level to pass to zstd or to > > leave it as default. I think Unit193 had some thoughts on that, so > > CC'ing him in case he wants to jump in. > > Yes. My initial version did bump the compression level as well, but in the > end I decided against that in order to give the user the option to control > that via ZSTD_CLEVEL. Good idea... So we could go with `xz -T0` to enable multi-threading (as this can not be controlled via environment variable). Then we document in man page how to influence compression level via environment variable. We have to set the environment variable from web server, no? Do we also want to document GZIP, XZ_DEFAULTS & friends? -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);} -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20200226/dbd549dd/attachment.asc> ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH v2 1/2] ui-snapshot: add support for zstd compression 2020-02-26 8:21 [PATCH 1/2] ui-snapshot: add support for zstd compression list 2020-02-26 8:21 ` [PATCH 2/2] tests: add tests for xz compressed snapshots list 2020-02-26 9:16 ` [PATCH 1/2] ui-snapshot: add support for zstd compression Jason @ 2020-02-26 21:24 ` list 2 siblings, 0 replies; 8+ messages in thread From: list @ 2020-02-26 21:24 UTC (permalink / raw) From: Christian Hesse <mail at eworm.de> This patch adds support for zstd [0] compressed snapshots (*.tar.zst). We enable multiple working threads (-T0) and compression level 10 (-10) for a trade-off between speed and file size. [0] https://www.zstd.net/ Signed-off-by: Christian Hesse <mail at eworm.de> --- cgitrc.5.txt | 5 +++-- tests/setup.sh | 2 +- tests/t0107-snapshot.sh | 42 +++++++++++++++++++++++++++++++++++++++++ ui-snapshot.c | 7 +++++++ 4 files changed, 53 insertions(+), 3 deletions(-) diff --git a/cgitrc.5.txt b/cgitrc.5.txt index 4ad3e64..b0ec698 100644 --- a/cgitrc.5.txt +++ b/cgitrc.5.txt @@ -407,8 +407,9 @@ side-by-side-diffs:: snapshots:: Text which specifies the default set of snapshot formats that cgit generates links for. The value is a space-separated list of zero or - more of the values "tar", "tar.gz", "tar.bz2", "tar.xz", "tar.lz" and - "zip". The special value "all" enables all snapshot formats. + more of the values "tar", "tar.gz", "tar.bz2", "tar.lz", "tar.xz", + "tar.zst" and "zip". The special value "all" enables all snapshot + formats. Default value: none. source-filter:: diff --git a/tests/setup.sh b/tests/setup.sh index 69e47e6..e09f7c5 100755 --- a/tests/setup.sh +++ b/tests/setup.sh @@ -104,7 +104,7 @@ virtual-root=/ cache-root=$PWD/cache cache-size=1021 -snapshots=tar.gz tar.bz tar.lz zip +snapshots=tar.gz tar.bz tar.lz tar.zst zip enable-log-filecount=1 enable-log-linecount=1 summary-log=5 diff --git a/tests/t0107-snapshot.sh b/tests/t0107-snapshot.sh index a845ad9..95ad624 100755 --- a/tests/t0107-snapshot.sh +++ b/tests/t0107-snapshot.sh @@ -80,6 +80,48 @@ test_expect_success LZIP 'verify untarred file-5' ' test_line_count = 1 master/file-5 ' +if test -n "$(which zstd 2>/dev/null)"; then + test_set_prereq ZSTD +else + say 'Skipping ZSTD validation tests: zstd not found' +fi + +test_expect_success ZSTD 'get foo/snapshot/master.tar.zst' ' + cgit_url "foo/snapshot/master.tar.zst" >tmp +' + +test_expect_success ZSTD 'check html headers' ' + head -n 1 tmp | + grep "Content-Type: application/x-zstd" && + + head -n 2 tmp | + grep "Content-Disposition: inline; filename=.master.tar.zst." +' + +test_expect_success ZSTD 'strip off the header lines' ' + strip_headers <tmp >master.tar.zst +' + +test_expect_success ZSTD 'verify zstd format' ' + zstd --test master.tar.zst && + cp master.tar.zst /tmp/. +' + +test_expect_success ZSTD 'untar' ' + rm -rf master && + tar --zstd -xf master.tar.zst +' + +test_expect_success ZSTD 'count files' ' + ls master/ >output && + test_line_count = 5 output +' + +test_expect_success ZSTD 'verify untarred file-5' ' + grep "^5$" master/file-5 && + test_line_count = 1 master/file-5 +' + test_expect_success 'get foo/snapshot/master.zip' ' cgit_url "foo/snapshot/master.zip" >tmp ' diff --git a/ui-snapshot.c b/ui-snapshot.c index 92cde42..fddf989 100644 --- a/ui-snapshot.c +++ b/ui-snapshot.c @@ -91,6 +91,12 @@ static int write_tar_xz_archive(const char *hex, const char *prefix) return write_compressed_tar_archive(hex, prefix, argv); } +static int write_tar_zstd_archive(const char *hex, const char *prefix) +{ + char *argv[] = { "zstd", "-T0", "-10", NULL }; + return write_compressed_tar_archive(hex, prefix, argv); +} + const struct cgit_snapshot_format cgit_snapshot_formats[] = { /* .tar must remain the 0 index */ { ".tar", "application/x-tar", write_tar_archive }, @@ -98,6 +104,7 @@ const struct cgit_snapshot_format cgit_snapshot_formats[] = { { ".tar.bz2", "application/x-bzip2", write_tar_bzip2_archive }, { ".tar.lz", "application/x-lzip", write_tar_lzip_archive }, { ".tar.xz", "application/x-xz", write_tar_xz_archive }, + { ".tar.zst", "application/x-zstd", write_tar_zstd_archive }, { ".zip", "application/x-zip", write_zip_archive }, { NULL } }; ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-02-26 22:38 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-02-26 8:21 [PATCH 1/2] ui-snapshot: add support for zstd compression list 2020-02-26 8:21 ` [PATCH 2/2] tests: add tests for xz compressed snapshots list 2020-02-26 9:19 ` Jason 2020-02-26 9:16 ` [PATCH 1/2] ui-snapshot: add support for zstd compression Jason 2020-02-26 10:18 ` list 2020-02-26 10:28 ` list [not found] ` <alpine.DEB.2.22.394.2002261715550.3793@Zeta> 2020-02-26 22:38 ` list 2020-02-26 21:24 ` [PATCH v2 " list
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).