From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) by hurricane.the-brannons.com (Postfix) with ESMTPS id 29D9F77DC1 for ; Tue, 29 Dec 2015 11:16:42 -0800 (PST) Received: by mail-wm0-x236.google.com with SMTP id u188so19673143wmu.1 for ; Tue, 29 Dec 2015 11:17:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:subject:message-id:mime-version:content-type :content-disposition:user-agent; bh=tDPzj0bc1dO3+QHhlU/lXR17PTvBQNbw4TxT5Eb1dVg=; b=0xxm5Il+2smnJLhY7sHAGB7Xliqf172GP67fu4ySkr8U/6crz8l9YfgJOAVzVmx1s0 I346UEZS9G+3qjUPiNX8M0R1ZP2iA8zda9/Z3R5unsEL+fyR58xBMAzbl2bIb2rSOolf lIKpWeTbtJucIKqTycT4PTYMniwM2MIdWqq5nwQwGdVR+FFXQaMynZzZPNVxOhh7ihGO HRAEccjE9hoRJ4T55od7ix8+fu9qb/VTF8GLBIXnYtJyBrVflcygKsECosff3c4Vv6bW NA4koGmaGmxARCCBmVba9mdcNkuep5eJzKKuTq3DQt563WU2dQs1wU6ed/EMpATDhEi9 7q4A== X-Received: by 10.28.132.74 with SMTP id g71mr5213187wmd.100.1451416625232; Tue, 29 Dec 2015 11:17:05 -0800 (PST) Received: from 122oven.adamthompson.me.uk (c.2.2.f.4.7.e.f.f.f.d.1.4.2.2.0.2.4.0.9.2.4.1.1.0.b.8.0.1.0.0.2.ip6.arpa. [2001:8b0:1142:9042:224:1dff:fe74:f22c]) by smtp.gmail.com with ESMTPSA id p9sm6800905wjy.41.2015.12.29.11.17.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 29 Dec 2015 11:17:04 -0800 (PST) Date: Tue, 29 Dec 2015 19:17:02 +0000 From: Adam Thompson To: Edbrowse-dev@lists.the-brannons.com Message-ID: <20151229191702.GB1766@122oven.adamthompson.me.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="u5E4XgoOPWr4PD9E" Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) Subject: [Edbrowse-dev] curl handles and general comms design X-BeenThere: edbrowse-dev@lists.the-brannons.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Edbrowse Development List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2015 19:16:42 -0000 --u5E4XgoOPWr4PD9E Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I noticed in a recent commit that we're now not keeping curl handles around. We probably don't want to head in this direction since that also means we're not doing persistant connections or correct things with cookies. I also notice that Edbrowse appears to be performing *extremely* poorly (ye= ah I know we're looking for functionality first, but still...). I'm not sure if the two things are related or not but it prompted me to hav= e a closer look at curl. I've listed some of what I found below, along with how I think we *could* use it. First of all, it seems the curl devs and I and Karl (I think?) were thinking along the same lines in terms of single-threaded multiplexed = curl transfers. To accomplish this there's a curl-multi interface which allows multiple concurrent transfers with either an event-driven or fd-based (read select o= r=20 poll) API. What this basically means is that we can add curl handles to a stack and th= en use an event loop to call curl_multi_perform and curl will handle all the multiplexing for us, telling us when things need attention. This also gets us connection sharing and dns cache sharing. To solve the parallel curl handles accessing cookie databases issue, there's also the curl-shared interface. I believe this can be used with the curl-multi interface since the curl-mul= ti interface is single-threaded so no need for mutexes etc. This allows us to share cookies between curl handles as well as other data should we need them. What this all means I think is that, by combining both interfaces, we should be able to create a single-threaded, essentially async, comms lay= er. This should also fix some of the strange cookie issues we have, as well as allow us to better use persistant connections and dns caching. I'm not sure if I'll have time to actually do the coding for this or not, but I think it's worth discussing. Any thoughts? Cheers, Adam. --u5E4XgoOPWr4PD9E Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWgtwuAAoJELZ22lNQBzHO0Y0IAJKBVixsdDUFomedtvS710x7 RWQPjHRqnKP3bupWcWMJKp8sQuwK1114IFfesF+Fz5+AcpLMehFDn/xeMcQdzxS5 0IeUSkLtrKoxA2/VW8Z2fAGHzScChxA9Hsc/R1+vN3f7ZNYWZB9wwzBYBYYa1skB ljMF6+D136K+e1+Gsg5PxRso8nEDN5I+m5aZuTtFtY58R34YfhYjb34hNFgb0frK a4m30BpBN9bi5XjAAFetgEOKvRL6UH7tV9NCcAOJpxcx95eD0vepKp0hswIERoZw 0uNJ97IM9BBNx5CO10e4X3lpArW9jh0MQj4gzJKZM40NKr5hIcZHkxtXslvzTaY= =b5ES -----END PGP SIGNATURE----- --u5E4XgoOPWr4PD9E--