[Edbrowse-dev] A couple of feature requests

[Edbrowse-dev] A couple of feature requests

[Adam Thompson]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

Hi all,

Today was my 2nd day in my new job, and thus my 2nd day of trying systems with
chris's static build of edbrowse (setting up the build environment on my work
machine's tonight's task). The systems I'm trying to access are
industry-standard, and I need to use a proxy to access the internet.

This brings me on to my first question/feature request:
As far as I'm aware there's no way in edbrowse to support proxy exceptions, i.e. hosts which shouldn't go via a specified proxy.  How difficult would this be to add?

The other issue I've been having is that one of the systems we use sends 401
responses with a login form, not authentication headers (I've never seen this
before either, and yes it breaks the standard). At the moment, as far as I can work out,
it looks like when I abort the edbrowse auth prompt it doesn't continue to
parse the response. Is it possible to change this so that in the case of an
aborted auth prompt (and better still if it can automagically happen in case of
a 401 status with no www-authenticate header),
the rest of the response is parsed? It may even be better to parse the response
anyway then display the authentication prompt if auth headers are sent,
but I'm not sure how possible that is.
Anyway, according to what I can read about 401 statuses,
there should be some way for the user to get to the body of the response as it
may be helpful in the case of a failed auth attempt.

If someone can tell me where to look,
I'm willing to help get both of these implemented.

Cheers,
Adam.
PS: Thanks Chris for the static build, it saved me much time yesterday.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Edbrowse-dev] A couple of feature requests

[Karl Dahlke]

In much of edbrowse, and even in some of life,
I try not to let the best be the enemy of the good.
The best would be the javascript pac file approach, and honestly I think
we should implement that in the near future.
I don't think it would be hard, given that we have javascript suppport,
and the underlying js function isn't likely to be complicated.
But sooner than that, perhaps we should get some noprox exceptions running.
That's even easier, and it's a stepping stone,
determining the proxy at connect time rather than init time.
And we must remember that sometimes we run edbrowse with js turned off,
perhaps in general or perhaps per site.
In that case the js context isn't even there.
We don't have the option of running the pac file.
I think some folks will want a little more proxy control even with js turned off.
Then, when js is enabled, and if there is a pac file,
we can support that.
So I propose starting with some proxy exceptions in the config file,
and seeing where that leads.

Karl Dahlke

Re: [Edbrowse-dev] A couple of feature requests

[Adam Thompson]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

On Thu, Jul 03, 2014 at 08:39:14AM -0700, Chris Brannon wrote:
> Karl Dahlke <eklhad@comcast.net> writes:
> 
> > Can you specify proxy exceptions within curl?
> 
> Nope.
> This will be a fun can of worms.
> In corporate settings, the proxy you use may depends on the
> URL you're trying to fetch.
> See http://en.wikipedia.org/wiki/Proxy_auto-config
> It's going to be more complicated than just supporting exceptions.

Whereas full PAC file support would be nice (and I'd like to see it in the
future), at my $large_company at the moment I think I only need a sufficiently
useful proxy selection mechanism. At the end of the day,
if we can get a sufficiently powerful mechanism in place then we can do the js
bindings to it.

That being said, I'm all in favour of designing the mechanism properly so that
supporting PAC files is just a case of a few js bindings.
Also, doing it this way we can support both hard-coded and PAC file based proxy selection, which would be really useful in my opinion.

Cheers,
Adam.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Edbrowse-dev] A couple of feature requests

[Chris Brannon]

Karl Dahlke <eklhad@comcast.net> writes:

> If we had set a proxy on the last http fetch, can we then unset it,

I'm sorry for missing the rest of your question.  Yes, setting
CURLOPT_PROXY to "" should do the trick, and we should be able to change
it whenever we need to.

-- Chris

Re: [Edbrowse-dev] A couple of feature requests

[Chris Brannon]

Karl Dahlke <eklhad@comcast.net> writes:

> Can you specify proxy exceptions within curl?

Nope.
This will be a fun can of worms.
In corporate settings, the proxy you use may depends on the
URL you're trying to fetch.
See http://en.wikipedia.org/wiki/Proxy_auto-config
It's going to be more complicated than just supporting exceptions.
Here's a long quote from that article:

<quote>
Computer {operating systems} (e.g., {Microsoft Windows}
{Mac OS X}
{Linux}) require a number of settings to communicate over the
{Internet}. These settings are typically obtained from an {Internet
Service Provider} (ISP). Either anonymous (proxy to use a {proxy
server}) or real settings may be used to establish a network connection.

The PAC File{edit}

The Proxy auto-config file format was originally designed by {Netscape}
in 1996 for the {Netscape Navigator 2.0}^{[1]} and is a
{text file} that defines at least one JavaScript function,
FindProxyForURL(url, host), with two arguments:
url is the URL of the object and host is the host-name derived from that URL.
By convention, the PAC file is normally named proxy.pac.
The {WPAD standard} uses wpad.dat.

To use it, a PAC file is published to a {HTTP server},
and client user agents are instructed to use it,
either by entering the URL in the proxy connection settings of the browser or
through the use of the WPAD protocol.

A very simple example of a PAC file is:

function FindProxyForURL(url, host)
{
	return "PROXY proxy.example.com:8080; DIRECT";
}


This function instructs the browser to retrieve all pages through the proxy on
{port} 8080 of the server proxy.example.com.
Should this proxy fail to respond, the browser contacts the Web-site directly,
without using a proxy. The latter may fail if {firewalls},
or other intermediary network devices,
reject requests from sources other than the proxy;
a common configuration in corporate networks.

A more complicated example demonstrates some available JavaScript functions to
be used in the FindProxyForURL function:

function FindProxyForURL(url, host) {
	// our local URLs from the domains below example.com don't need a proxy:
	if (shExpMatch(host, "*.example.com"))
	{
		return "DIRECT";
	}

	// URLs within this network are accessed through
	// port 8080 on fastproxy.example.com:
	if (isInNet(host, "10.0.0.0", "255.255.248.0"))
	{
		return "PROXY fastproxy.example.com:8080";
	}

	// All other requests go through port 8080 of proxy.example.com.
	// should that fail to respond, go directly to the WWW:
	return "PROXY proxy.example.com:8080; DIRECT";
}
</quote>

Yeah, I had to deal with this stuff when I worked for $big_company.  For
all intents and purposes, edbrowse was completely unusable on that
network.

-- Chris

Re: [Edbrowse-dev] A couple of feature requests

[Adam Thompson]

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

On Thu, Jul 03, 2014 at 05:19:51AM -0400, Karl Dahlke wrote:
> > As far as I'm aware there's no way in edbrowse to support proxy exceptions,
> 
> Chris you've done more curling than the rest of us.
> Can you specify proxy exceptions within curl?
> If not, and I'm guessing not,
> then we have to move the PROXY curl call from my_curl_init()
> into httpConnect, so it can be done per host.
> If we had set a proxy on the last http fetch, can we then unset it,
> or set it to null or "" or some such, so as not to use any proxy
> on this fetch? Will that work?
> If yes then I could implement this one without a lot of fuss.
> I may start by setting up noprox in the config file,
> the way we have nojs now.
> That would build the list of exceptions, which is the needed infrastructure.

That'd be brilliant, especially if like the nojs setup we could have an
exception for an entire domain.

As a bonus it'd be nice if we supported different proxys for different protocols (like curl, links and wget do).
I would also ask for SOCKS proxy support but that's probably a
step too far at the moment.

> I'll have to think about your 401 request, there's more going on there.

Yeah, I suspected as much. Tbh I was quite surprised to find a system that did
this (provide a 401 with no www-authenticate), but other browsers (chrome, firefox etc) handle it automatically,
which is probably why the developers didn't realise (or didn't care)
that it was non-standard behavior.

Cheers,
Adam.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Edbrowse-dev] A couple of feature requests

[Karl Dahlke]

> As far as I'm aware there's no way in edbrowse to support proxy exceptions,

Chris you've done more curling than the rest of us.
Can you specify proxy exceptions within curl?
If not, and I'm guessing not,
then we have to move the PROXY curl call from my_curl_init()
into httpConnect, so it can be done per host.
If we had set a proxy on the last http fetch, can we then unset it,
or set it to null or "" or some such, so as not to use any proxy
on this fetch? Will that work?
If yes then I could implement this one without a lot of fuss.
I may start by setting up noprox in the config file,
the way we have nojs now.
That would build the list of exceptions, which is the needed infrastructure.

I'll have to think about your 401 request, there's more going on there.

Karl Dahlke