Re: [Edbrowse-dev] curl handles and general comms design

[Adam Thompson <arthompson1990@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4280 bytes --]

On Tue, Dec 29, 2015 at 02:57:27PM -0500, Karl Dahlke wrote:
> Yes, sorry if edbrowse is in a state of flux right now.
> I'm closing curl handles more as a proof of concept,
> not saying we need to or want to do that in general.

That's ok, I understand that. I'm running the latest dev version, and thus,
as I've said in the past I think, things'll probably break.
What had me a little worried was the amount of possible breakages (and the damn
near 30 second page load times for the Jenkins CI system we use at work, as well as a bunch of other sites).

> Can we retain our cookie state with at least one handle open?

From reading the docs, it appears as if, as long as the share handle is open the cookies are retained, though this may not be quite correct.

> We know about the sharing interface and Chris is going to work on that,
> and then all the handles should tie together with one cookie state
> and then some things should clear up.
> Some of the inefficiency could be rereading and rewriting the cookie file
> all the time, and that will go away.

Yes, that makes sense.

> I put more of our state variables for downloading files etc
> on the stack in preparation for threads.
> I use to use static variables and got away with it
> since processes forked off, but we know that won't work
> so now moving to stacks in preparation for threads.
> This is the "curl http background download mini project"
> that we agreed on last week,
> and it should finish up soon and then we can talk about the next step.

As I said in my last email, I don't think we even need threads for this,
just curl-multi with it's single threaded multiplexing.
I work with an architecture similar to this and it genuinely works very
scalably, with buffering and callbacks ensuring you get multiplexed transfers.
Given how well-tested curl is, I've no reason to doubt their scalability claims
and I really think that this will work.

> I'm thinking about an edbrowse-http process to handle these curl requests
> and maintain the one and only cookie state, but perhaps not just for edbrowse,
> perhaps for all instances of edbrowse running.
> I run edbrowse from various virtual consoles, sometimes without realizing it,
> and indeed one edbrowse could clobber cookies introduced by another edbrowse.
> I haven't noticed that specifically but it probably has happened.

Agreed.

> So if edbrowse-http became a daemon serving the curl needs
> of all edbrowse processes running,
> then that would solve any and all cookie collision problems forever more.
> Wonder if Chrome and others effectively share the cookie jar and favorites etc
> among separately running instances of the browser?
> Well it's something to think about.

Yes they do by default as far as I know.
I think I'd probably call it something like edbrowse-curl though and then add
in all the curl stuff we do since, as far as I can work out,
the multi interface (not sure about the share interface but quite possibly)
can handle different protocols concurrently.

> I also need to resurrect some of the socket routines that are buried in git,
> so we can use them.
> They're suppose to hide the differences between unix and windows,
> and they were tested at one time but that was like 15 years ago,
> so I'll want Geoff to look at them again.

Yeah, things've probably changed a bit since then.

> Send any bug reports or disasters to me,
> and hopefully this will stabilize soon.

Ok, if I can I'd like to actually do some coding on edbrowse again,
but I'm not sure if we'd just end up with massive conflicts.

Also, thinking about background and foreground downloads,
if we go down the curl-multi route or the separate comms process actually
(single-threaded via curl-multi or multi-threaded...),
then background and foreground (though not memory vs file unfortunately)
simply becomes a question of whether we print dots when we receive a certain
amount of data or not. Since edbrowse will be reading from the comms process
(or from curl events) that could even be toggleable during downloads since the
dot printing and the user would be using the same thread and process. It'd just be a question of setting a flag somewhere.

Regards,
Adam.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]