"musl" gzip-compressed tarballs are decompressed if using newest wegt release

"musl" gzip-compressed tarballs are decompressed if using newest wegt release

[Jens Schleusener]

Hi,

not a real "musl" software problem but just an observation of a changed 
behaviour of "musl" tarball downloads with the newest "wegt" release 
(1.19.2): The tarballs are saved now decompressed but contains the 
original extension (.tar.gz). Although if the extension would be .tar it 
seems at least to me an undesired behaviour.

Reason seems a new "wget" feature, here some extracted lines taken from 
the ChangeLog

  2017-08-04
  Add gzip Content-Encoding decompression
  (gethttp): Decompress files with gzip Content-Encoding

in combination with the HTTP header the www.musl-libc.org server delivers:

  Content-Type: application/x-tar
  Content-Encoding: gzip

Solutions/workarounds may be on the server side delivering of a HTTP 
header like

  Content-Type: application/x-gzip
  (or Content-Type: application/octet-stream)

or on the client side the use of the new "wget" option

  --compression=none

Regards

Jens

Re: "musl" gzip-compressed tarballs are decompressed if using newest wegt release

[Rich Felker]

On Wed, Nov 01, 2017 at 11:19:30AM +0100, Jens Schleusener wrote:
> Hi,
> 
> not a real "musl" software problem but just an observation of a
> changed behaviour of "musl" tarball downloads with the newest "wegt"
> release (1.19.2): The tarballs are saved now decompressed but
> contains the original extension (.tar.gz). Although if the extension
> would be .tar it seems at least to me an undesired behaviour.
> 
> Reason seems a new "wget" feature, here some extracted lines taken
> from the ChangeLog
> 
>  2017-08-04
>  Add gzip Content-Encoding decompression
>  (gethttp): Decompress files with gzip Content-Encoding
> 
> in combination with the HTTP header the www.musl-libc.org server delivers:
> 
>  Content-Type: application/x-tar
>  Content-Encoding: gzip
> 
> Solutions/workarounds may be on the server side delivering of a HTTP
> header like
> 
>  Content-Type: application/x-gzip
>  (or Content-Type: application/octet-stream)
> 
> or on the client side the use of the new "wget" option
> 
>  --compression=none

IMO this is both a bug in the server and a major regression in wget,
since lots of servers have this bug. I'll look at fixing it on my side
but I think it should be reported to wget and reverted since this is
going to break a *huge* number of build scripts that download tarballs
from affected servers.

Rich

Re: "musl" gzip-compressed tarballs are decompressed if using newest wegt release

[Jens Schleusener]

On Wed, 1 Nov 2017, Rich Felker wrote:

> On Wed, Nov 01, 2017 at 11:19:30AM +0100, Jens Schleusener wrote:
>> Hi,
>>
>> not a real "musl" software problem but just an observation of a
>> changed behaviour of "musl" tarball downloads with the newest "wegt"
>> release (1.19.2): The tarballs are saved now decompressed but
>> contains the original extension (.tar.gz). Although if the extension
>> would be .tar it seems at least to me an undesired behaviour.
>>
>> Reason seems a new "wget" feature, here some extracted lines taken
>> from the ChangeLog
>>
>>  2017-08-04
>>  Add gzip Content-Encoding decompression
>>  (gethttp): Decompress files with gzip Content-Encoding
>>
>> in combination with the HTTP header the www.musl-libc.org server delivers:
>>
>>  Content-Type: application/x-tar
>>  Content-Encoding: gzip
>>
>> Solutions/workarounds may be on the server side delivering of a HTTP
>> header like
>>
>>  Content-Type: application/x-gzip
>>  (or Content-Type: application/octet-stream)
>>
>> or on the client side the use of the new "wget" option
>>
>>  --compression=none
>
> IMO this is both a bug in the server and a major regression in wget,
> since lots of servers have this bug. I'll look at fixing it on my side
> but I think it should be reported to wget and reverted since this is
> going to break a *huge* number of build scripts that download tarballs
> from affected servers.

Thanks for the more clear statement (I agree). If not already done by 
another person I will send an accoding mail to the "bug-wget" list.

Jens

Re: "musl" gzip-compressed tarballs are decompressed if using newest wegt release

[Rich Felker]

On Wed, Nov 01, 2017 at 04:50:01PM +0100, Jens Schleusener wrote:
> On Wed, 1 Nov 2017, Rich Felker wrote:
> 
> >On Wed, Nov 01, 2017 at 11:19:30AM +0100, Jens Schleusener wrote:
> >>Hi,
> >>
> >>not a real "musl" software problem but just an observation of a
> >>changed behaviour of "musl" tarball downloads with the newest "wegt"
> >>release (1.19.2): The tarballs are saved now decompressed but
> >>contains the original extension (.tar.gz). Although if the extension
> >>would be .tar it seems at least to me an undesired behaviour.
> >>
> >>Reason seems a new "wget" feature, here some extracted lines taken
> >>from the ChangeLog
> >>
> >> 2017-08-04
> >> Add gzip Content-Encoding decompression
> >> (gethttp): Decompress files with gzip Content-Encoding
> >>
> >>in combination with the HTTP header the www.musl-libc.org server delivers:
> >>
> >> Content-Type: application/x-tar
> >> Content-Encoding: gzip
> >>
> >>Solutions/workarounds may be on the server side delivering of a HTTP
> >>header like
> >>
> >> Content-Type: application/x-gzip
> >> (or Content-Type: application/octet-stream)
> >>
> >>or on the client side the use of the new "wget" option
> >>
> >> --compression=none
> >
> >IMO this is both a bug in the server and a major regression in wget,
> >since lots of servers have this bug. I'll look at fixing it on my side
> >but I think it should be reported to wget and reverted since this is
> >going to break a *huge* number of build scripts that download tarballs
> >from affected servers.
> 
> Thanks for the more clear statement (I agree). If not already done
> by another person I will send an accoding mail to the "bug-wget"
> list.

It should be fixed now. I've written a patch for thttpd that removes
the content-transfer-encoding nonsense and submitted it for inclusion
in Alpine Linux (used on musl's web host) and already installed the
patched package myself. Hopefully this will eventually make it
upstream to thttpd, but it seems mostly unmaintained.

FWIW I'd love to switch to a better httpd that's actually maintained,
but I need something with simple migration path and without loads of
unwanted bloat. Needs to be able to:

- support host header based vhosts, translated to dir names under web
  root.

- execute cgi (only!) from a configured whitelist of pathnames

- support url paths that go past the cgi program, i.e. /cgit/musl/...
  where /cgit is the cgi executable

- honor symlinks (without any attempt to [re]interpret them)

Solution which involve a process per static-content request are not
really desirable. Recommendations would be welcome.

Rich

Re: "musl" gzip-compressed tarballs are decompressed if using newest wegt release

[Matias Fonzo]

On Wed, 1 Nov 2017 14:08:07 -0400
Rich Felker <dalias@libc.org> wrote:

> On Wed, Nov 01, 2017 at 04:50:01PM +0100, Jens Schleusener wrote:
> > On Wed, 1 Nov 2017, Rich Felker wrote:
>
> [..]
>
> FWIW I'd love to switch to a better httpd that's actually maintained,
> but I need something with simple migration path and without loads of
> unwanted bloat. Needs to be able to:
> 
> - support host header based vhosts, translated to dir names under web
>   root.
> 
> - execute cgi (only!) from a configured whitelist of pathnames
> 
> - support url paths that go past the cgi program, i.e. /cgit/musl/...
>   where /cgit is the cgi executable
> 
> - honor symlinks (without any attempt to [re]interpret them)
> 
> Solution which involve a process per static-content request are not
> really desirable. Recommendations would be welcome.
> 

Take a look to http://www.hiawatha-webserver.org/