From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) by hurricane.the-brannons.com (Postfix) with ESMTPS id 9CF1077D0D for ; Sat, 26 Dec 2015 01:03:00 -0800 (PST) Received: by mail-wm0-x235.google.com with SMTP id p187so220519691wmp.0 for ; Sat, 26 Dec 2015 01:03:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=52X3lMvHDt0052Xl0/lxD23AZooD76vbnY33ynvUg+A=; b=CBvlr7ii6kUX3qBUQnc7R9fILHApKZQ2IyCY021J3phu0t1WGIlykr7AIJTw2rN/9y ScnjlahliZUWhE0diml2ffkkzWrQ0j7bY1SWkvUXF9gvOEmUh6HTIX9SlT0LDKS1cBh7 6Pc8ngr/iOX8cTdDmH6z0KR9jmGScSzUrLHuYF4rlNH/vtnLiiKCV+zuezMemX3N0zMl fc8ySXc7vFrB1kVKCOdLWL5KXEMBYt/2oDP5PseJTpKXvVcNo6EN6nV5T5WJzvhmLjAa RKO5h6sKSll/HJ4BOMD6EZXAQeY+qKpnts9lP78GopsH1A5dANnf0N9bA4EpzfABwV3T tBag== X-Received: by 10.194.117.68 with SMTP id kc4mr4265131wjb.111.1451120596307; Sat, 26 Dec 2015 01:03:16 -0800 (PST) Received: from hob.adamthompson.me.uk (40.77.155.90.in-addr.arpa. [90.155.77.40]) by smtp.gmail.com with ESMTPSA id kb4sm21986221wjb.41.2015.12.26.01.03.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 26 Dec 2015 01:03:15 -0800 (PST) Date: Sat, 26 Dec 2015 09:03:05 +0000 From: Adam Thompson To: Karl Dahlke Cc: Edbrowse-dev@lists.the-brannons.com Message-ID: <20151226090305.GA3144@hob.adamthompson.me.uk> References: <20151126004524.eklhad@comcast.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="n8g4imXOkfNTN/H1" Content-Disposition: inline In-Reply-To: <20151126004524.eklhad@comcast.net> User-Agent: Mutt/1.5.24 (2015-08-30) Subject: Re: [Edbrowse-dev] curl thread test X-BeenThere: edbrowse-dev@lists.the-brannons.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Edbrowse Development List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Dec 2015 09:03:01 -0000 --n8g4imXOkfNTN/H1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 26, 2015 at 12:45:24AM -0500, Karl Dahlke wrote: > In follow up to my previous post on processes and threads, > I propose the following small project / test. >=20 > At present, http or ftp can download in the background, at least on unix. > This is useful for large files or slow internet or not a lot of ram etc. > It works by forking the process and reissuing the http request. > This is harmless 99% of the time, but even here you wonder > if the separate processes and separate curl instances could mismatch. > You go on to look at 3 more websites during the download, more cookies ar= e added to the jar, > then the download finishes and that process writes out the jar > and clobbers the new cookies. > Since I don't really understand when curl writes the jar it's hard to say > whether this could happen or not. Neither do I tbh, it seems like it could though, unless curl uses some sort of shared memory if one shares a curl handle acr= oss processes. > With this in mind, how bout this modest project. >=20 > A download determined to be in the background > spins up a thread, rather than forking a process. > The thread runs the http request and when that is done > it sets, in the appropriate BG_JOB structure, state =3D 0, > then the thread exits. > Seems manageable, and it has the advantage of working on windows, > which currently does not support the download in background feature at al= l. > Also, and more important, we can test the threadsafe nature of curl. > While downloading a large file in background, pull up some other small we= b pages. > Do these all run in their threads, and does the whole assemblage > still use one cookie space, > making transient cookies available to all threads and consistently writin= g the cookie jar? > We really need to know this. Yeah, that sounds like a good idea. In fact, I'm wondering how possible it would be to make all network stuff spin up threads by default, but then choose whether we want to background the threa= d or not. The only parts we need to make thread safe for this are the curl buffers and that job structure. > It's incremental, and as Adam writes, > it increments us in the right direction. > What do you think? I think if this works then it'll certainly help with a lot of things. I'd still quite like to have a shared comms process in the end and keep js = in a separate process, but I was thinking of multi-threading the comms stuff any= way. So that works out as 3 processes for the moment, and fixes the issues of js and curl with parallel cookie jar writing etc. I'm going to look into IPC on Windows (though can't test it) to see if there is *anything* we can do to make this happen. Cheers, Adam. --n8g4imXOkfNTN/H1 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWflfJAAoJELZ22lNQBzHO4zYH/j9VrlayurDvj4+HrYf4PIol X351cNspsfiiTvWzM12HlEcijLMlVuRUrCYKfxtQgppsuzCK9cCCUstYFPG3wkNr 93DJNlKNMPl0oZum9muYVFNLKErKa7AsMtKCSf7oxp0CimUo/fZn6UeQ+aGHTy1D 4n6lOxGMPmZX/lzJMkViIFnb7nvGtH1z43WTsO/vK9RgHB34xzp2rJqx2YYCOSox P0S5AvZVre+xXvIOfTfPp1mWB/Pi1RX52+GegfOhff7/TVcuYF0KGdVHtFnL96pi 8T/dRPS52qAH5/2aSlxdXme9zfTtGlg//o0L7tlogQoc+/g24Ej1px2HToXfwdk= =xsHQ -----END PGP SIGNATURE----- --n8g4imXOkfNTN/H1--