From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) by hurricane.the-brannons.com (Postfix) with ESMTPS id 299FB77D0D for ; Wed, 30 Dec 2015 04:01:16 -0800 (PST) Received: by mail-wm0-x22f.google.com with SMTP id l65so57909473wmf.1 for ; Wed, 30 Dec 2015 04:01:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=/l6U3cWQLn3U3yRGddq0MfFCkVrnDFFlKjpPKBuGD9E=; b=AxdisHIFo2o8NvaxmRnzqn7YAh7ijp7vqaLHjPuuQnMdqvRmmUW87kyYYgJISGnlVi Z6amvFZsrZT6bJ7BZHSq5OulpyWYuDK0+5FnU6kM71ZU+z4Tl8JmNR+vBriEyEK6CTeV KPrGZCipgxGnocXWg4Dfn505pyZHTTMViyO2EM0XLbOe/MHWOMMZgrWoIBInJnrnEZbA OTnb36g60V0hBz6tCrHy1Yfr83DzAzVBWW9EZ9UV1ewjtuudxsrInLw1s6TNuRBQGn22 FxeoKGg/igyfgD1J6YBDIzZxGzxSi2CUydzbk336yx69mRP2QEZbjgTvOHAnI0QU8+p0 fuNQ== X-Received: by 10.194.103.198 with SMTP id fy6mr59538207wjb.48.1451476900844; Wed, 30 Dec 2015 04:01:40 -0800 (PST) Received: from 122oven.adamthompson.me.uk (c.2.2.f.4.7.e.f.f.f.d.1.4.2.2.0.2.4.0.9.2.4.1.1.0.b.8.0.1.0.0.2.ip6.arpa. [2001:8b0:1142:9042:224:1dff:fe74:f22c]) by smtp.gmail.com with ESMTPSA id yz5sm13350346wjc.36.2015.12.30.04.01.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 30 Dec 2015 04:01:40 -0800 (PST) Date: Wed, 30 Dec 2015 12:01:34 +0000 From: Adam Thompson To: Karl Dahlke Cc: Edbrowse-dev@lists.the-brannons.com Message-ID: <20151230120123.GA2471@122oven.adamthompson.me.uk> References: <20151229191702.GB1766@122oven.adamthompson.me.uk> <20151129145727.eklhad@comcast.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="32u276st3Jlj2kUU" Content-Disposition: inline In-Reply-To: <20151129145727.eklhad@comcast.net> User-Agent: Mutt/1.5.24 (2015-08-30) Subject: Re: [Edbrowse-dev] curl handles and general comms design X-BeenThere: edbrowse-dev@lists.the-brannons.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Edbrowse Development List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 12:01:16 -0000 --32u276st3Jlj2kUU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Dec 29, 2015 at 02:57:27PM -0500, Karl Dahlke wrote: > Yes, sorry if edbrowse is in a state of flux right now. > I'm closing curl handles more as a proof of concept, > not saying we need to or want to do that in general. That's ok, I understand that. I'm running the latest dev version, and thus, as I've said in the past I think, things'll probably break. What had me a little worried was the amount of possible breakages (and the = damn near 30 second page load times for the Jenkins CI system we use at work, as= well as a bunch of other sites). > Can we retain our cookie state with at least one handle open? =46rom reading the docs, it appears as if, as long as the share handle is o= pen the cookies are retained, though this may not be quite correct. > We know about the sharing interface and Chris is going to work on that, > and then all the handles should tie together with one cookie state > and then some things should clear up. > Some of the inefficiency could be rereading and rewriting the cookie file > all the time, and that will go away. Yes, that makes sense. > I put more of our state variables for downloading files etc > on the stack in preparation for threads. > I use to use static variables and got away with it > since processes forked off, but we know that won't work > so now moving to stacks in preparation for threads. > This is the "curl http background download mini project" > that we agreed on last week, > and it should finish up soon and then we can talk about the next step. As I said in my last email, I don't think we even need threads for this, just curl-multi with it's single threaded multiplexing. I work with an architecture similar to this and it genuinely works very scalably, with buffering and callbacks ensuring you get multiplexed transfe= rs. Given how well-tested curl is, I've no reason to doubt their scalability cl= aims and I really think that this will work. > I'm thinking about an edbrowse-http process to handle these curl requests > and maintain the one and only cookie state, but perhaps not just for edbr= owse, > perhaps for all instances of edbrowse running. > I run edbrowse from various virtual consoles, sometimes without realizing= it, > and indeed one edbrowse could clobber cookies introduced by another edbro= wse. > I haven't noticed that specifically but it probably has happened. Agreed. > So if edbrowse-http became a daemon serving the curl needs > of all edbrowse processes running, > then that would solve any and all cookie collision problems forever more. > Wonder if Chrome and others effectively share the cookie jar and favorite= s etc > among separately running instances of the browser? > Well it's something to think about. Yes they do by default as far as I know. I think I'd probably call it something like edbrowse-curl though and then a= dd in all the curl stuff we do since, as far as I can work out, the multi interface (not sure about the share interface but quite possibly) can handle different protocols concurrently. > I also need to resurrect some of the socket routines that are buried in g= it, > so we can use them. > They're suppose to hide the differences between unix and windows, > and they were tested at one time but that was like 15 years ago, > so I'll want Geoff to look at them again. Yeah, things've probably changed a bit since then. > Send any bug reports or disasters to me, > and hopefully this will stabilize soon. Ok, if I can I'd like to actually do some coding on edbrowse again, but I'm not sure if we'd just end up with massive conflicts. Also, thinking about background and foreground downloads, if we go down the curl-multi route or the separate comms process actually (single-threaded via curl-multi or multi-threaded...), then background and foreground (though not memory vs file unfortunately) simply becomes a question of whether we print dots when we receive a certain amount of data or not. Since edbrowse will be reading from the comms process (or from curl events) that could even be toggleable during downloads since = the dot printing and the user would be using the same thread and process. It'd = just be a question of setting a flag somewhere. Regards, Adam. --32u276st3Jlj2kUU Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWg8eeAAoJELZ22lNQBzHOF98IAM41aodi1T21KMWP3KpAImC9 tWtiPZv+VrHIc1QEpiAU/tGBGQqrjza25N669d5HUREeacle2SZQWEIuytHn4GG1 gPz7UIBSdMmiUxoDa3XcPpi5ukrKkpgGztqu7IFJckgP14Qv5PXNmTuH6qT29rhL GYvQLT8zQmYJqZrRThwbDpczoLU3G7wEE2a1vk9DTXkklDYySpUEXsb9tUgAN+Ai 7BR196Ra3CrlvRDtNTsYKnpFt+BeRoNT8NVbrjwZmONgmhyO/sdcRK4ZfOeHeQbA OTdE04Z6766j7RwywvUx8InNHmJjqssaxu/pPQzrILdxRzvzV2vDjJ/0f/mck6A= =zL/a -----END PGP SIGNATURE----- --32u276st3Jlj2kUU--