[PATCH 1/2] ui-snapshot: add support for zstd compression

[list at eworm.de (Christian Hesse)]

"Jason A. Donenfeld" <Jason at zx2c4.com> on Wed, 2020/02/26 17:16:
> We had been discussing which compression level to pass to zstd or to
> leave it as default. I think Unit193 had some thoughts on that, so
> CC'ing him in case he wants to jump in.

Thought about that myself after I sent the mail...

This is a trade-off... Do we prefer faster speed or better compression? We
can not have both, just either of both or a mix.
And do we want to go with multi-threaded compression?

The default compression level is 3. For our Arch packages we decided to go
with level 20... Well, the packages are compressed once and decompressed on a
magnitude of machines - so it makes sense there.

I would propose to take something in between... I think compression level 10
gives reasonable results. Any thoughts?

Some numbers:

% for I in $(seq 3 19); do zstd -$I < git-2.25.1.tar > git-2.25.1-$I.tar.zst;
done
level 3: 0,30s user 0,03s system 105% cpu 0,306 total
level 4: 0,30s user 0,03s system 107% cpu 0,307 total
level 5: 0,57s user 0,03s system 103% cpu 0,574 total
level 6: 0,72s user 0,04s system 103% cpu 0,739 total
level 7: 0,90s user 0,04s system 102% cpu 0,912 total
level 8: 1,13s user 0,03s system 102% cpu 1,134 total
level 9: 1,59s user 0,02s system 100% cpu 1,598 total
level 10: 1,98s user 0,04s system 100% cpu 2,002 total
level 11: 2,28s user 0,05s system 100% cpu 2,318 total
level 12: 3,50s user 0,05s system 100% cpu 3,536 total
level 13: 4,70s user 0,05s system 100% cpu 4,749 total
level 14: 6,52s user 0,06s system 99% cpu 6,603 total
level 15: 7,06s user 0,09s system 100% cpu 7,146 total
level 16: 9,35s user 0,10s system 99% cpu 9,456 total
level 17: 12,00s user 0,08s system 99% cpu 12,118 total
level 18: 14,26s user 0,08s system 99% cpu 14,390 total
level 19: 19,66s user 0,10s system 99% cpu 19,835 total
% for I in $(seq 3 19); do zstd -T0 -$I < git-2.25.1.tar >
git-2.25.1-$I-T0.tar.zst; done
level 3: 0,41s user 0,05s system 182% cpu 0,251 total
level 4: 0,52s user 0,05s system 200% cpu 0,285 total
level 5: 0,81s user 0,04s system 195% cpu 0,434 total 
level 6: 1,59s user 0,06s system 188% cpu 0,877 total
level 7: 1,25s user 0,04s system 189% cpu 0,681 total
level 8: 1,63s user 0,04s system 187% cpu 0,888 total
level 9: 2,12s user 0,05s system 187% cpu 1,156 total
level 10: 2,58s user 0,07s system 178% cpu 1,482 total
level 11: 2,85s user 0,07s system 176% cpu 1,650 total
level 12: 4,33s user 0,12s system 173% cpu 2,560 total
level 13: 5,50s user 0,07s system 188% cpu 2,948 total
level 14: 6,42s user 0,09s system 192% cpu 3,389 total
level 15: 8,21s user 0,13s system 185% cpu 4,492 total
level 16: 10,28s user 0,07s system 179% cpu 5,778 total
level 17: 12,08s user 0,10s system 117% cpu 10,411 total
level 18: 16,24s user 0,11s system 114% cpu 14,271 total
level 19: 24,78s user 0,21s system 125% cpu 19,843 total

% xz < git-2.25.1.tar > git-2.25.1.tar.xz
17,07s user 0,26s system 139% cpu 12,453 total
% time gzip < git-2.25.1.tar > git-2.25.1.tar.gz
2,16s user 0,04s system 361% cpu 0,607 total

 37M git-2.25.1.tar
8,7M git-2.25.1.tar.gz
5,9M git-2.25.1.tar.xz
8,2M git-2.25.1-3.tar.zst
8,1M git-2.25.1-4.tar.zst
7,9M git-2.25.1-5.tar.zst
7,6M git-2.25.1-6.tar.zst
7,3M git-2.25.1-7.tar.zst
7,1M git-2.25.1-8.tar.zst
7,0M git-2.25.1-9.tar.zst
6,8M git-2.25.1-10.tar.zst
6,8M git-2.25.1-11.tar.zst
6,7M git-2.25.1-12.tar.zst
6,5M git-2.25.1-13.tar.zst
6,4M git-2.25.1-14.tar.zst
6,4M git-2.25.1-15.tar.zst
6,1M git-2.25.1-16.tar.zst
6,0M git-2.25.1-17.tar.zst
5,9M git-2.25.1-18.tar.zst
5,9M git-2.25.1-19.tar.zst
-- 
main(a){char*c=/*    Schoene Gruesse                         */"B?IJj;MEH"
"CX:;",b;for(a/*    Best regards             my address:    */=0;b=c[a++];)
putchar(b-1/(/*    Chris            cc -ox -xc - && ./x    */b/42*2-3)*42);}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20200226/3ad5fb8b/attachment.asc>