bug: webfs can't download files from sourceforge.net

bug: webfs can't download files from sourceforge.net

[Nick Owens]

executing the following command:

echo useragent curl/7.28.1 > /mnt/web/ctl
hget https://download.sourceforge.net/tinyscheme/tinyscheme-1.41.tar.gz
> tinyscheme-1.41.tar.gz

will produce a html file.

using actual curl on unix works fine.

the problem appears to be the "Referer" header that webfs forced to be
attached to the request. removing the Referer header from the webfs
source and executing the request again produces the expected .tar.gz
file.

to me the correct solution seems to be to simply drop Referer headers
from webfs, and add them back to mothra if need be, but the comment
attached to the Referer-adding code makes me worried about making a
hasty change.

any thoughts?

Re: bug: webfs can't download files from sourceforge.net

[Nick Owens]

after a long and grueling discussion on irc, Referer headers have been
removed from webfs. if anyone encounters a problem with a website as a
result of this, feel free to complain.

fixed in commit 3fe2d53fd3d8.
On Tue, Sep 18, 2018 at 12:31 PM Nick Owens <mischief@offblast.org> wrote:
>
> executing the following command:
>
> echo useragent curl/7.28.1 > /mnt/web/ctl
> hget https://download.sourceforge.net/tinyscheme/tinyscheme-1.41.tar.gz
> > tinyscheme-1.41.tar.gz
>
> will produce a html file.
>
> using actual curl on unix works fine.
>
> the problem appears to be the "Referer" header that webfs forced to be
> attached to the request. removing the Referer header from the webfs
> source and executing the request again produces the expected .tar.gz
> file.
>
> to me the correct solution seems to be to simply drop Referer headers
> from webfs, and add them back to mothra if need be, but the comment
> attached to the Referer-adding code makes me worried about making a
> hasty change.
>
> any thoughts?

Re: [9front] Re: bug: webfs can't download files from sourceforge.net

[Kurt H Maier]

On Tue, Sep 18, 2018 at 01:47:09PM -0700, Nick Owens wrote:
> after a long and grueling discussion on irc, Referer headers have been
> removed from webfs. if anyone encounters a problem with a website as a
> result of this, feel free to complain.

Given the fact that several major browsers have instituted plans to
strip all pathinfo from referer headers or abnegate them entirely, it
would be more appropriate to complain to the operater of the broken
website than anywhere else.

khm

Re: [9front] Re: bug: webfs can't download files from sourceforge.net

[Ethan Gardener]

On Tue, Sep 18, 2018, at 10:20 PM, Kurt H Maier wrote:
> On Tue, Sep 18, 2018 at 01:47:09PM -0700, Nick Owens wrote:
> > after a long and grueling discussion on irc, Referer headers have been
> > removed from webfs. if anyone encounters a problem with a website as a
> > result of this, feel free to complain.
> 
> Given the fact that several major browsers have instituted plans to
> strip all pathinfo from referer headers or abnegate them entirely, it
> would be more appropriate to complain to the operater of the broken
> website than anywhere else.

I agree, with only a little regret.  There's this one barely-maintained site, 
but I never use it in Mothra any more.