edbrowse-dev - development list for edbrowse
 help / color / mirror / Atom feed
* [edbrowse-dev] Another curl question
@ 2019-09-03 20:10 Karl Dahlke
  2019-09-04 14:08 ` Adam Thompson
  0 siblings, 1 reply; 4+ messages in thread
From: Karl Dahlke @ 2019-09-03 20:10 UTC (permalink / raw)
  To: edbrowse-dev

This happens on my relatively old machine and on a newer machine.

In edbrowse and with db4

e http://iyfsearch.com/px.js?ch=2

It asks if you want to download, just hit space to pull it into memory.
It is a regular fetch and it goes into cache. No problem.

Quit edbrowse and bring it up again. db4 and do the same thing.

e http://iyfsearch.com/px.js?ch=2

It sees it in cache and just does a head request.
The server should give you the same etag and you pull it out of cache, but,
curl says "could not read the data from the server"
Thus the file is not fetched at all
It looks like we get all the right stuff from the server.

What gives?
What happens when other browsers issue head requests?

Karl Dahlke

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [edbrowse-dev] Another curl question
  2019-09-03 20:10 [edbrowse-dev] Another curl question Karl Dahlke
@ 2019-09-04 14:08 ` Adam Thompson
  2019-09-04 15:03   ` Karl Dahlke
  0 siblings, 1 reply; 4+ messages in thread
From: Adam Thompson @ 2019-09-04 14:08 UTC (permalink / raw)
  To: Karl Dahlke; +Cc: edbrowse-dev

On Tue, Sep 03, 2019 at 04:10:36PM -0400, Karl Dahlke wrote:
> This happens on my relatively old machine and on a newer machine.
> 
> In edbrowse and with db4
> 
> e http://iyfsearch.com/px.js?ch=2
> 
> It asks if you want to download, just hit space to pull it into memory.
> It is a regular fetch and it goes into cache. No problem.
> 
> Quit edbrowse and bring it up again. db4 and do the same thing.
> 
> e http://iyfsearch.com/px.js?ch=2
> 
> It sees it in cache and just does a head request.
> The server should give you the same etag and you pull it out of cache, but,
> curl says "could not read the data from the server"
> Thus the file is not fetched at all
> It looks like we get all the right stuff from the server.
> 
> What gives?

On my machine (Debian sid updated as of... a few days ago I think):
curl --head -vv http://iyfsearch.com/px.js?ch=2
*   Trying 208.91.196.46:80...
* TCP_NODELAY set
* Connected to iyfsearch.com (208.91.196.46) port 80 (#0)
> HEAD /px.js?ch=2 HTTP/1.1
> Host: iyfsearch.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

curl --version
curl 7.65.3 (x86_64-pc-linux-gnu) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11
libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.39.2
librtmp/2.3
Release-Date: 2019-07-19
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3
pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile
libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

> What happens when other browsers issue head requests?

Dunno, but my guess is this site'll reset the connection on them as well...
very odd.  However, in my experience (my day job is developing a web
security service) there are some very odd HTTP implementations out there.
My guess is that server simply cannot handle HEAD requests and, rather than
speaking correct HTTP, just drops the connection.  That being said, it may
be worth trying with a different user agent (I've seen that kind of oddness
before as well).

Cheers,
Adam.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [edbrowse-dev] Another curl question
  2019-09-04 14:08 ` Adam Thompson
@ 2019-09-04 15:03   ` Karl Dahlke
  2019-09-04 15:08     ` Adam Thompson
  0 siblings, 1 reply; 4+ messages in thread
From: Karl Dahlke @ 2019-09-04 15:03 UTC (permalink / raw)
  To: edbrowse-dev

> My guess is that server simply cannot handle HEAD requests and, rather than

Right, so, I now revert back to get upon curl head failure; I use to just give up.
Least this way you get your data, even if it didn't come from cache.

Karl Dahlke

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [edbrowse-dev] Another curl question
  2019-09-04 15:03   ` Karl Dahlke
@ 2019-09-04 15:08     ` Adam Thompson
  0 siblings, 0 replies; 4+ messages in thread
From: Adam Thompson @ 2019-09-04 15:08 UTC (permalink / raw)
  To: Karl Dahlke; +Cc: edbrowse-dev

On Wed, Sep 04, 2019 at 11:03:26AM -0400, Karl Dahlke wrote:
> > My guess is that server simply cannot handle HEAD requests and, rather than
> 
> Right, so, I now revert back to get upon curl head failure; I use to just give up.
> Least this way you get your data, even if it didn't come from cache.

That'll work and sounds sensible.  I also wonder about the possibility of exposing the curl debug messages, perhaps at some debug level.

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2019-09-04 15:08 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-03 20:10 [edbrowse-dev] Another curl question Karl Dahlke
2019-09-04 14:08 ` Adam Thompson
2019-09-04 15:03   ` Karl Dahlke
2019-09-04 15:08     ` Adam Thompson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).