From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/8414 Path: news.gmane.org!not-for-mail From: Jens Gustedt Newsgroups: gmane.linux.lib.musl.general Subject: Re: spin strategy in __wait Date: Sun, 30 Aug 2015 17:31:39 +0200 Message-ID: <1440948698.693.21.camel@inria.fr> References: <1440875657.693.12.camel@inria.fr> <20150830005745.GI7833@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-3MtxOcsq7Me6T4zlxxYi" X-Trace: ger.gmane.org 1440948769 12787 80.91.229.3 (30 Aug 2015 15:32:49 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 30 Aug 2015 15:32:49 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-8426-gllmg-musl=m.gmane.org@lists.openwall.com Sun Aug 30 17:32:43 2015 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1ZW4b4-0007zi-G5 for gllmg-musl@m.gmane.org; Sun, 30 Aug 2015 17:32:43 +0200 Original-Received: (qmail 24499 invoked by uid 550); 30 Aug 2015 15:32:39 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 24474 invoked from network); 30 Aug 2015 15:32:38 -0000 X-IronPort-AV: E=Sophos;i="5.17,435,1437429600"; d="scan'";a="175355658" In-Reply-To: <20150830005745.GI7833@brightrain.aerifal.cx> Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAABHNCSVQICAgIfAhkiAAAEb9JREFUaN7VmXuQ3lV5xz/n/O7v7333fffdW3azSTY3Qi6wGy4aEc0qWlGhWcAq7WiDrWNpO61paS29EpzWaevYpvfL2JK0VRBbBxQtg0U2BRGKhIAI4ZKwSTbZ+3u//G7nnP7BtGM7UyVAnfb56/xznuf7nXnOnOf5fuH/eYjXI8mDn/5k8ch90xNj+eKka8xkI1MTGmY6Rk/XpD5w091fPvnf7xz8sQ/sLTQ6k54SY2mW1mQY3DVX8Kdv/PvPnfyBE7hn797bxBPP3NBv28RAqgyzJqNgJIs5q7bhHW+94Yd+5/fuBrjj5l/cs/LUs/vNiVMTo5aNJQX9lks3iaivHqlt/9D7xs778Y/Uf6AE/uWa6x7wT52ZTDyLKEkoOHmW4oSaSDFxQmq7tHPuDJ5TUkv10iAGgSLvediZotf16HS6KG1RuXDT/g99/vO3vtLa8vUgELuCVAja0mVJKRaylGbgYnwP27bImYRcozkmKrWS53vYjosfhNSUIjXQUoq2K0ksRdZqTZ5Lbfs/Dl/66E/uSebmb8hF8URab48JS+IPDVJtto9uuHj8wKW//6lD/1OSznxtLEPRv2kdPH2MSBpkqlDGkLoOIgGkIWcEiTYsk1JSDpGUFAAcBzfRCKXxV+ql7/XWnn7uxcm3vOGS2rrR1UcLV+6pC4B//tjPjGdHjhwttFOEcMmMAgFeLkezWsOTkmigfyYZGTzYN7HtwFt u+rU6wJnprxQf/JvP7beffH5fBIwGIZqEutJUlCKSNoFjYbQiihOkFmih0FJgSUGIpGxstE6RwiLQmsyzcXZesO8N1773YOHKPfUv/uxP726ePjsVNVoT9WZjMq02kRpEKay9+UPXjQmAL1991UuDZ+fGQulSVQrpuqg4pWkkupSjVGshtGZJpzQ9DysfQqZIkgzR7 jAc+qAFrm2xTIRJJKnr0okTfCnQKLRlo9MUicGWL7dcnCnytoOTKTxpERpBv8moomm7AZFjEddb9EiHdhJhPJdYanQKUZrivmH7Pnlm+itFuVgdi7WkpSSJdFlqRwRugLAkQakXJQ06cAhcB0tnRI0mstOllKYEvk8qJJGj8IygID16hwZBGIzIsIXCRREq6DGSsu9RkBYBBldKbCFwbMnA2mH8wKIrbQrGY1WkKbQ6DFkWOQEicMks8JTEQRJI0LOLJbvxwosTMlMk0kYbjTCGci4gzrr0Sod0YQFHWiSuRTHMkbRaCCxMZlh2FLGQLEjBmHFIhUBrEI0OlgYnyNN0LFKliYxGeT69rouqtOjP5+lLEtxugosgOjNHj7ZRGmwjaDoGX9skKEQQIFptkAIjIWckCQIrE9ija8aOznk5wiwh9lw8aeMlMamUeEKis4wOFiaFRtpEYxBJSoQA4ZM4ksgSPG4E9Siibjt02xUaUQsnzFFfSZlPM4TRlPp7yQWCs/PzlMs53HrMRLGPt/f2UFYRbkOzbGm6JkPgYjAgbVo6I/Ntup0unuPgWAK0RaLVy//AfVe864lgcWkicz1yloNvoJMmpFLTsiCJNIFt0VERHdcnGyxPW5598BmlJo6tVPc9v1RB2CHdqMWOYpkLCwHnhyHVTHNcpTxUrVCp1rAtyYZ1w0i3SLFk89A3/g2UZH3O4ubL30RxsU5ldhkpDFJIbGFRE4rYSLQQRHGEJQQeFm7gYHaev98CuO6NF62q1hqTKktJlKarNW0JDZUgHA9dys/Y q4emnQ3Df3nxj7z7+t2f+pO/Ysv5U0dm525uKYty3wCetPjRdYNcjaaQabq1Oq1MkVOKkc1reGFhiWIuj7C6JGlKyZNsNor3n7eFStLlkcoKz61fvX/79u3T9cWliTBWvrQEHa0QQmIZgdIK27FQmaKVxIRbN+77z5/4sT/+1PjJEycmosWVmXJvP3YpZM3aNYyuWz dTuHLPf5lPrnr3VXtPnDp10HNDlIqROEwO9fG+VosXspQ7lyto3+fE8ln2rlpDfniQP332WRCCd75tB43jZ7jUDRlspTiugxAeR+I2960s0hBy3503fmT69FcfONian5uQliDIF+m2YyKlUUIRxRF968eOfuC+r+4851Hill//+PiRx79ztNrs0mm0KZaLZEnGWwsB725WOFRv8HAmSDodSo7D3oF+bMfirxbOYkKHQDhc5lusznxaaO6fO8O2fIE9vf1M0+Go0jM73nT55EcvGa8duedr+3S1si8v7VLnzDxRpkjCHGJ4aHr1utVT7/mzP6/b5wJ++v57izf9wi8dVNrHcT3yxQK27RJ3Y6zM0LI1b+0ZoFldIvFsxvsH6LMELQOpkST1GBVq3NIAvvC54ztPcTwRLOs2b+sf4u2RS5JFY1++5yt3XbXnvZPX3/n5W4FbH/nLA+ueuOeBG/zefG3ynW87+N3D3jkRuP3226fiVEwkOiV0bNIsJep28QOfqOAjl2DEElw1WCZKFGUtkK6h7fj4DpRKfURJjeGwSE/UJcJga4d6p0lHCoo64w1BkfsqtYnf/LVPHgSuAdh1476TwMsD3t9/9jUMc9pMSiGxJRgV0+22Wa7XqTfqHH7pON8u9/NE6BB5PnZPD42BkMe14IG4gVXIsVSvkijFU2fPIrOUraPrsD3Jlt4SnTQlsi2qtqLSarG0VJnaObHztu8HyTqX9vn6Aw/cMTw0TJpkeH6AhaIncCn3FkmFzWNzc3y7U+ObjTqPVJs8Vq1h1oScabdoV xWxyQg9l57BATZZDpuCgH5PMrVxM3G3g513WfNDW3liYRGsPN1ua2LHtq0zM6dOPfmqCDx15227/+ILd58EaNa717/00smp3bsvo7fcS3VpGTcAKV2q1QZJ3MVIC88JsFJN2c+xbrDM+aHPlqE+ji0vY7kWvaFhaaXJzs3nYaoNNq/tJ1/tsiwtnpEOtz/+HHNLMXG UkmYp7XarVKlWD53TQtO89+7iN//stoPWqdkpPx/Wur3F6WccZ+KFdnuskXNZaCVkiWJ++Qy+59EbuHi6y1t2XkL27Ak2CJssS6lHMYPCYDLDP/dJvj1fQ0qb+WoMqk3BzrFlYAidc5hZXAQnh9DQ6TbRQhF3U7Is4eZf+YWJj9308Se/5z7w3fH8s8fGnNOnp1ZJie52S+VOd2oIeJObo2Vs5i2JWZWn1e/TrjQZK/WSiyPCE2eImh1S12PJaEJtYbsCN81oL7eROPQVodVI6ZiA1MvxQhyhOxGWlcOkKUHg4aQOaaoQ0hDFXf7pi3dPAk+e00r56HuurubnzpZsY5NKiYMhwUagcY1CCINB0pACIQXLoaBHWdTbXbTwWHEhHwQE9SbVgRJ/Mz/Hi4sr7H3fOIuVJvc+OEsYFnG8AKETGs0WljBYtk2j3nkZmRDoqMPqdSNHHztyZOc5rZTpQHlfIh2MpdDSgAZbZ/hGgzRIaSEF9AiNLyWylRFbNh0bfKEZsCxkK2JxZJi/Pv0CJyornLd5DQ8++hxHv7NIEFg0W00ajRqNRh3XtciUot5okBlFZgzSssm04fjxExN/+kcHxs+JwOWHDh06u3bkQNV1a7ZSIDWehEQoUgEaQGtMlpHlLXp6XPLdjFyqwBMsAvfR5D4Z0fFKbNqwgWajgnQHWF6K8V0H35EIrQFodWKSzGAJQc63sS2JylKkZaO1ZnTNqplXpUo07727+NV/vPuAv3D2hrWpRtYaZJbEwmBlINf00144S35kgMZsxEpgc0zEfL3Spu 1LklihlU2u6NBtNZDaohXHIDKadYWRLsZkZFhgNCaLcB2XzECWalAZhR6v9vzx472v+BF/dxSu3FPf9cY3z0TNLgO+YFNfHzozJHFE4CvW1CPGBvtYabQ5YQuON+aI7YCqUTSX2hgTUeop025IstSiv1wiUQu4fkiz2SBTHaSQ9AQOaZqBk6MTdzFGYAnB0FAvWjD9 fVWJ7xUjI+XJ54/NMNeVLM3XMEZjWRYGzf2dWUp5kJZNmlmE+RDdaXHpznFOzc6xOL8AKIqlArVaTKdTQSvD4sIifb29KOOgshaOJal0UzSgjcQYyNKU4zMzXHTR+F2vSRdKOm3CnIvnO4QFn77ePL7vErVaqLhN4OUYcGHL+j7iVoeBoX5mT5+mtjzHeRtLDA2UWFqYx3c8Ot2MocE875y8iMCHLElIogytDVJKjFIYLUiiCKVihkeGD379Xx869JoICGHj2C4qU2ASFqsLNFptvMBj9ZoSlzgen7hiFz919QQDoWB+9gwqU4SFXk6eXGHN6kH6e21Wls7SX3Zot5t889HvUKlFL8ssJkOgAIXv2+gswqiYjRvHjv7qr+zb94qEre8Vnu/juRlKCTJlgQhBxdRaTVb1lNh7yRb41tOUR99G72Afza6h0e7gezZWOMzRY7NIZVHoyVOtxli2hSUEaarJlwLqOqOTaoTMYVmagXKOQnHk6O/+7u9MTl5xZf01D3NrR0dK7W58pe/naHS7SB1z7fg6fm58I2/OuRSencH2HIIffjNzLRhfv5ZGo85itU6aSRr1Jq6laEcJjcgwWA7YMOoS5B2WKx38oEAUJURRhko1nm9P/8EffvrK7wf+FbfQjT/zUwdzoTvTjVv4UlK0XHZs3UTZd1mbaqyiCwKc02d4v51ycW2Zy9Zvw+AQ5DxW9RcoFwRDZZ9VA2UAdu26iD3vupQLto5SrXSwpIPjKsbWr+aDH/zg/lcC/pzU6Y98eO/4iReem67WOyU/6KFdX2S kELLRU3y0d5Cg1aEZBqSxQJdtprXL5546guXm6QtCwmKOTrvC2NoSq8tw8fYSDx9t8tATVSqNOo16lzdech7Ndrz/a4e/9fqr05+57dCTW7edtz/fE2JIySyHlxqKr8+3OFXooykND8ocf1vwuD1WHO1G5AKPUi5EYVGpLeA4OcqhZtVASKtjkLZFpjM6nZjNG0dnN p63fepcwL8qf+BdV+x+YGmxNimRtNOUJI3Z1APX7NjBPxx5ntjLQRLTTBKGSz4jwz3UajWGhgap1uqcPFXh0ovX88zTz1Mo9tJNQdiF6d273zh1y29/un6ueM7ZH7hw24apfGDPJN02ri0phXlUaZi/e3qGSpziOgKjI8qhS73eQKuUIBAsLS3i+x5DQ0W6zRUyY1hpiNrAwND+PXve+6rAv2qH5rdv+dXxr933tbtWlqpj/cOjkAkWaosUHUW5UGBmYZF1AwM0WjUKvX2g25TLJZ56dh7Pztg5vmFm8vIL9udHLrtr6trrXxXw12wxTd9/b/ETt95y15lTC5Pbtu2gUVkgaTeItSJKDI4Fo6t66cZtbNuiUu8QlvpnVq0auuGzd3758P8Jl3L6/nuLf3Tg05OWEJOFfH4iiuLJOMlYXmnSrDURaIxJEFJzwYXb9x+640u38v8hPvazP717+5atZtvmrWbN6vVm9dBqc83VP7znf6OW/Xok+a2b941XG9HY7KmTE81me/Khww+P5XMFkszQiFIKPQEQTwF3/58zui+98IK9KA4W8gVOn10kF4Tk8z7SdWl2I6JWA0fE9A+VqdXq0x/+iQ/f8PM3/cbJ14vAa7JZr3rXFR+bn188aFk2veUCxXIPQeiDkdQbbXKuZP3qIju3rcVKm7z9su2TTxy+a+Yzn3z/La8XAevVXrzu2j3jBnlXrVonVSmtbpdms40AwlyeTGeEtk2r00HYLhdsXc/OrR4XbVaU/Mbkb930nqme0rrowUefffIHTuC6a/fsPfbM C3d0WolfKhTp6clRyhcg04xtGCHt1Dh/rExRamYWGyxXa7xn1yYuvkAwP7fClg0Zhx+qr/rM7d+Yesc73nHD5OW7Zh478tRzPxACH/iRa245duzkAdsJfMuy8aXm/FUFlpeWWOm2WZidI9WapaUlFIaTlQ4jfR6/vO9D2NE32LTeYnZOUmnA6PAohx9+qnTy1NnrP/ Cj15cu23XZI4/826Px/xqB37z547sPH374oOvnKTsOb+1zuCoImZARM8ZiLlKkXcWOcoFRJ+Do/CwFO4e2DEe+9Ti+ZXA8ny/cU+X4bMKRZxZIVEiqXI6/+NKuUzPHb7z40ovnX3zx+Ctuq38HyuqWG7Tu+A0AAAAASUVORK5CYII= X-Mailer: Evolution 3.12.9-1+b1 Xref: news.gmane.org gmane.linux.lib.musl.general:8414 Archived-At: --=-3MtxOcsq7Me6T4zlxxYi Content-Type: multipart/mixed; boundary="=-uIGXMJ/Ke6Af2zSjv47N" --=-uIGXMJ/Ke6Af2zSjv47N Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Am Samstag, den 29.08.2015, 20:57 -0400 schrieb Rich Felker: > On Sat, Aug 29, 2015 at 09:14:17PM +0200, Jens Gustedt wrote: > > The question that I asked myself is how much spinning to we need in > > such lock/wait primitives. In the first attached graphic, the three > > bottom curves show what difference three spinning strategies can make > > for this test on a 2x2 hyperthreaded core machine. The bottom is no > > spinning at all, the next is the current strategy implemented in musl, > > and the third is a modification of that. As you can see, for a high > > load of threads they can make a substantial difference, but you also > > see that musl's actually strategy is not very different from doing no > > spinning at all. >=20 > That's intentional when you have waiters. Notice in your graph that > for the range 2-4 threads, performance in musl is comparable to your > "no-shortcut" approach. This shows that the spinning is working as > intended. In that range, yes. When the machine is overcommitted with, say, 2 times more threads than the number of cores, it is significantly off. > BTW I think the names you chose are misleading. :-) Refraining to spin > is not a "shortcut" musl is taking. It's not trying to be quicker, but > to refrain from aggressive behavior which should lead to pathological > lock unfairness. I would call the approaches aggressive-spin, > opportunistic-spin, and no-spin. Probably, they were really just "working" names for my scripts. > > The "new" strategy is simply to avoid to take the shortcut that the > > actual code takes when spinning. Here, currently when we detect that > > are already waiters, we stop spinning and try to go into futex_wait > > immediately. Where this sounds nice at a first glance, the figures > > seem to indicate that this is not a good idea and that we would be > > better off without. I'll send a patch for that in a next mail, and > > also one that lets you modify the number of spins easily. >=20 > I don't think you can say we'd be better off in general with a single > test. Sure, but it should be taken as a data point. Average behavior of the malloc implementation seems to be important to me. > In areas like this you need to be able to demonstrate that there > are no cases where the expected pathologically bad behavior can > happen. I suspect you get lucky with malloc because there's a fair > amount of code being executed between successive attempts on the lock > rather than intense hammering. Perhaps. But the behavior that I observe is not so much different where I used these things as locks for the atomic stuff. There the work that is done inside the critical section is just a memcpy/memcmp of 16 byte. > On the other hand, are you even counting the number of times each > individual thread performs a malloc/free cycle, or only the total > across all threads? Is it possible that, among your 256 threads, the > vast majority of malloc/free cycles are being performed by a small > number and the rest are stuck repeating futex_wait over and over? I now also collected these statistics, and, no, all threads get their share. I attach to data files (tab separated values) that show this. These files have one line per run (10 sec each), with the following columns number of threads average number per thread of calls to malloc/free during the 10 secs standard deviation of that quantity the min value over all threads the max value over all threads As you may see, the variation is the biggest around 3-4 threads, but stays reasonable, I think, from overall min to overall max there is a difference of 50% or so. Nobody is blocked. This variation then gets really small for up to 64 threads. For 128 and 256 it is a bit bigger, but overall the numbers are a bit too small for that cases and one should perhaps run the test for more than 10 secs. > > Now the only situation I thought of where this could be important is > > monoprocessors where actually spinning might not be so good and > > aborting it early good be necessary. So I rand the same test with > > taskset to nail the process to just one core. The result of that are > > the top two curves. As you can see, here the spinning strategy has > > almost no influence so I think we are safe to apply this patch. >=20 > This is very surprising to me, which further makes me doubt the test > methodology. The current number of spins is tuned to be comparable in > magnitude to the syscall cost, at least on x86. On my machine, x86_64, with are quite recent kernel it is not, it is a factor of 10 off. As I said, I observe a factor of 10 between a spin and a syscall, and not 100 as the existing code suggests. And, BTW, I also observed that the call to "a_spin" doesn't have much effect on my machine, neither positive nor negative. > Adding this extra cost > to each contended lock operation (you basically never stop spinning > early on single-core) should produce a measurable difference. No, on single core you have only one thread active at any time, and the probability that it is descheduled in the middle when holding the lock is tiny. So basically nobody is ever spinning as long as the time that is spend in the critical section is small. (And you may have sensible outlayers, once a thread is descheduled in the middle.) > > Now all of this can also be read as a performance test of the malloc > > subsystem, and here my feeling goes in the direction that Rich > > recently indicated. The performance of the "application" is much > > better if I eliminate all parallelism. As an additional indication > > there are to additional curves that fix the process to one core and > > its hyperthreaded sibling. > >=20 > > So maybe we might be better off with a simpler malloc strategy that > > serializes all requests. >=20 > I tried this a few weeks ago and it performed significantly worse on > my malloc stress tests. And it will get even worse as we try to scale > to larger numbers of cores, or worse yet NUMA... Did you try running your malloc stress test with taskset to nail the process to one core? Jens --=20 :: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS ::: :: ::::::::::::::: office Strasbourg : +33 368854536 :: :: :::::::::::::::::::::: gsm France : +33 651400183 :: :: ::::::::::::::: gsm international : +49 15737185122 :: :: http://icube-icps.unistra.fr/index.php/Jens_Gustedt :: --=-uIGXMJ/Ke6Af2zSjv47N Content-Disposition: attachment; filename="test-16b-spin0-per-thread.csv" Content-Type: text/csv; name="test-16b-spin0-per-thread.csv"; charset="UTF-8" Content-Transfer-Encoding: base64 CTEJMS4wMTAxNmUrMDgJLW5hbgkxLjAxMDE2ZSswOAkxLjAxMDE2ZSswOAoJMQkxLjA0NzMyZSsw OAktbmFuCTEuMDQ3MzJlKzA4CTEuMDQ3MzJlKzA4CgkxCTEuMDQ5MTRlKzA4CS1uYW4JMS4wNDkx NGUrMDgJMS4wNDkxNGUrMDgKCTEJMS4wNDk1MmUrMDgJLW5hbgkxLjA0OTUyZSswOAkxLjA0OTUy ZSswOAoJMQkxLjA1MDg2ZSswOAktbmFuCTEuMDUwODZlKzA4CTEuMDUwODZlKzA4CgkyCTIuNDYw MzllKzA3CTIuMTg5OTJlKzA2CTIuMzA1NTRlKzA3CTIuNjE1MjRlKzA3CgkyCTIuNTA5NjNlKzA3 CTIuMzk4NjllKzA2CTIuMzQwMDFlKzA3CTIuNjc5MjRlKzA3CgkyCTIuNTE1ODFlKzA3CTExODk4 OAkyLjUwNzM5ZSswNwkyLjUyNDIyZSswNwoJMgkyLjUyNDgyZSswNwkxODQxMjQJMi41MTE4ZSsw NwkyLjUzNzg0ZSswNwoJMgkyLjUzODc4ZSswNwk3NzE5OS44CTIuNTMzMzJlKzA3CTIuNTQ0MjRl KzA3CgkzCTEuNjc5NDFlKzA3CTEuMjI3NzdlKzA2CTEuNTUxMDNlKzA3CTEuNzk1NjllKzA3Cgkz CTEuNjkyOTllKzA3CTIuODg4NTJlKzA2CTEuMzc4MjFlKzA3CTEuOTQ1ODhlKzA3CgkzCTEuNjk3 NDZlKzA3CTEuMTEzODNlKzA2CTEuNTg2MTVlKzA3CTEuODA4OTFlKzA3CgkzCTEuNjk4MDNlKzA3 CTIuNzI1OTZlKzA2CTEuMzg5NDVlKzA3CTEuOTA2MDhlKzA3CgkzCTEuNzA2MTFlKzA3CTIuOTY5 NTllKzA2CTEuNDMwMDhlKzA3CTIuMDIwMzFlKzA3Cgk0CTEuMDYyNzhlKzA3CTU0OTcwLjUJMS4w NTc4NGUrMDcJMS4wNjk3NmUrMDcKCTQJMS4wNjU5OWUrMDcJMTQ4NzQuOQkxLjA2Mzg5ZSswNwkx LjA2NzMzZSswNwoJNAkxLjA2Nzc4ZSswNwk3NTQ0MC4zCTEuMDU4OThlKzA3CTEuMDc0NzJlKzA3 Cgk0CTEuMDcwMzJlKzA3CTMwOTg0MQkxLjAzNjQ4ZSswNwkxLjA5ODEyZSswNwoJNAkxLjA3MTU1 ZSswNwk1MjQ1Ni44CTEuMDY2NjllKzA3CTEuMDc4NmUrMDcKCTYJNi4yMzczOGUrMDYJNTIwMzUJ Ni4xNzI3NWUrMDYJNi4zMTUxMWUrMDYKCTYJNi45MDg1ZSswNgk1NzMwOS45CTYuODMzZSswNgk2 Ljk4NzQ0ZSswNgoJNgk2LjkxNTg1ZSswNgk1NzQzNS4zCTYuODA0MjFlKzA2CTYuOTU4MzhlKzA2 Cgk2CTYuOTIyNzhlKzA2CTQxNjQzLjcJNi44NTY2NGUrMDYJNi45ODYyNmUrMDYKCTYJNi45NTIx NGUrMDYJNDgzODQuOQk2LjkwNTEyZSswNgk3LjAxNTg1ZSswNgoJOAk1LjEyMTAzZSswNgkxODM2 Ny41CTUuMDkyMzVlKzA2CTUuMTM5NjVlKzA2Cgk4CTUuMTYxOTllKzA2CTIzMTU3LjQJNS4xMzQz MmUrMDYJNS4yMDMyMWUrMDYKCTgJNS4xNjUwOWUrMDYJNDI4MDgJNS4wODA3M2UrMDYJNS4yMTY2 MmUrMDYKCTgJNS4xNjkyMWUrMDYJMjI3NTMuMQk1LjExODQxZSswNgk1LjE4OTA4ZSswNgoJOAk1 LjE3NzcxZSswNgkyOTE5MC42CTUuMTMyNmUrMDYJNS4yMTE2NGUrMDYKCTEyCTMuMzcyNzdlKzA2 CTEzMzM2CTMuMzU1NDVlKzA2CTMuMzk4NDdlKzA2CgkxMgkzLjM3MzA4ZSswNgkxNzAzMS41CTMu MzQzOTRlKzA2CTMuNDA5MThlKzA2CgkxMgkzLjM4NDU3ZSswNgkxNzM3MQkzLjM0MDM0ZSswNgkz LjQwODI3ZSswNgoJMTIJMy4zODgzM2UrMDYJMTE4MjEuOQkzLjM3MTkzZSswNgkzLjQwNzA4ZSsw NgoJMTIJMy4zODkyZSswNgkxNTk1Mi40CTMuMzYzMzVlKzA2CTMuNDI0MzJlKzA2CgkxNgkyLjQ5 OTc1ZSswNgk4NjE3Ljk3CTIuNDg1MDllKzA2CTIuNTE1MDNlKzA2CgkxNgkyLjUwMzQ3ZSswNgkx MDUyMi40CTIuNDg1MjNlKzA2CTIuNTE5MTFlKzA2CgkxNgkyLjUwNTU1ZSswNgkxMTUxMgkyLjQ4 MzVlKzA2CTIuNTMwOTJlKzA2CgkxNgkyLjUxNTg4ZSswNgkxMDUwNC43CTIuNDk3MjdlKzA2CTIu NTMyNTNlKzA2CgkxNgkyLjUxODk2ZSswNgkxMzkzNC43CTIuNDgxNzFlKzA2CTIuNTQzMDhlKzA2 CgkzMgkxLjIxMjU0ZSswNgk3Njc1LjcyCTEuMTk3OTllKzA2CTEuMjI5MjJlKzA2CgkzMgkxLjIx MzE4ZSswNgk3MzAwLjEJMS4xOTMzMWUrMDYJMS4yMjcwMmUrMDYKCTMyCTEuMjEzMjZlKzA2CTkx MzYuNjIJMS4xOTIxNmUrMDYJMS4yMjk4ZSswNgoJMzIJMS4yMTY5N2UrMDYJOTgyMi4wOQkxLjIw MTk0ZSswNgkxLjI0MjE2ZSswNgoJMzIJMS4yMjEyNWUrMDYJMTE0NjEuNgkxLjIwMjIyZSswNgkx LjI0OTQ5ZSswNgoJNjQJNTAyMjYxCTQ0NTkuODMJNDg5MzMxCTUxMjY2NwoJNjQJNTg3ODc4CTUx ODcuNDgJNTY5MTA1CTYwMTA1MAoJNjQJNTg4MTU5CTUyNjQuNDkJNTc4NTk5CTYwMjAxNAoJNjQJ NTg4NDgzCTc5NzkuODEJNTcyNjQwCTYwOTA4NgoJNjQJNTg5NTE3CTY1MTQuNzYJNTc3NzMxCTYw OTIyMgoJMTI4CTI1MDIxMwk0MjY0LjcJMjQyMDIyCTI2NDU1OQoJMTI4CTI4NzA3MQk3MjExLjgy CTI3MjAzMQkzMTE2NTcKCTEyOAkyODgyOTMJNDUxMC4yOQkyNzgwNzcJMzAyMjgxCgkxMjgJMjg4 NjY3CTU3NTUuNjgJMjc2MjIxCTMwNjczNwoJMTI4CTI4ODczNgk2NDMwLjQJMjcyMjU3CTMwMTgz MAoJMjU2CTExNzgzNgkzNzU5Ljk5CTEwOTcwOQkxMzA1MjgKCTI1NgkxMTg1MzEJNTc0NS4zMgkx MDQ2NDIJMTM4OTkxCgkyNTYJMTM2ODA1CTY3NDQuMDgJMTE5ODMyCTE1NjUwMwoJMjU2CTEzNzMy NQk1MDcwLjU2CTEyNDMzNQkxNTA3MTcKCTI1NgkxMzkzNzIJNDI1Ni41MwkxMjc0MTgJMTUyMDc1 Cg== --=-uIGXMJ/Ke6Af2zSjv47N Content-Disposition: attachment; filename="test-16b-no-shortcut-per-thread.csv" Content-Type: text/csv; name="test-16b-no-shortcut-per-thread.csv"; charset="UTF-8" Content-Transfer-Encoding: base64 CTEJMS4wMjg0NWUrMDgJLW5hbgkxLjAyODQ1ZSswOAkxLjAyODQ1ZSswOAoJMQkxLjA0MjA2ZSsw OAktbmFuCTEuMDQyMDZlKzA4CTEuMDQyMDZlKzA4CgkxCTEuMDQ2MjFlKzA4CS1uYW4JMS4wNDYy MWUrMDgJMS4wNDYyMWUrMDgKCTEJMS4wNDcwMWUrMDgJLW5hbgkxLjA0NzAxZSswOAkxLjA0NzAx ZSswOAoJMQkxLjA0NzM5ZSswOAktbmFuCTEuMDQ3MzllKzA4CTEuMDQ3MzllKzA4CgkyCTIuNjQy NDFlKzA3CTUwOTg3MwkyLjYwNjM2ZSswNwkyLjY3ODQ2ZSswNwoJMgkyLjY0NDE1ZSswNwkxLjc3 NTI0ZSswNgkyLjUxODYyZSswNwkyLjc2OTY4ZSswNwoJMgkyLjY1MTY3ZSswNwkxMDM0MAkyLjY1 MDk0ZSswNwkyLjY1MjRlKzA3CgkyCTIuNjYxNThlKzA3CTU3ODcxNwkyLjYyMDY2ZSswNwkyLjcw MjVlKzA3CgkyCTIuNjc3MDJlKzA3CTExODkzNQkyLjY2ODYxZSswNwkyLjY4NTQzZSswNwoJMwkx LjgyMzY5ZSswNwkxLjUwOTExZSswNgkxLjcxMDJlKzA3CTEuOTk0OTZlKzA3CgkzCTEuODM0ZSsw NwkyLjEwNTM0ZSswNgkxLjY1NTA1ZSswNwkyLjA2NTk4ZSswNwoJMwkxLjgzNjE4ZSswNwk0LjUz NTM1ZSswNgkxLjU3MzkxZSswNwkyLjM1OTg3ZSswNwoJMwkxLjg1MzUyZSswNwkxLjAwOTkyZSsw NgkxLjc0Njc2ZSswNwkxLjk0NzU0ZSswNwoJMwkxLjkxMzA1ZSswNwkzLjAyMzMzZSswNgkxLjcx NjMxZSswNwkyLjI2MTE3ZSswNwoJNAkxLjIzMjg0ZSswNwk0NjMwNwkxLjIyNjA5ZSswNwkxLjIz NjQ0ZSswNwoJNAkxLjIzNDAxZSswNwk0NjAzNC43CTEuMjI4MDdlKzA3CTEuMjM5MDNlKzA3Cgk0 CTEuMjQ1NDJlKzA3CTIwNjEzNwkxLjIxNDg1ZSswNwkxLjI2ZSswNwoJNAkxLjI1NTQ1ZSswNwkx LjI2NjgxZSswNgkxLjA5Mjk5ZSswNwkxLjQwMjM2ZSswNwoJNAkxLjI5MDk5ZSswNwk0OTE5MS4y CTEuMjg2OTllKzA3CTEuMjk3MmUrMDcKCTYJOC4yMzAxMmUrMDYJMjgwODMxCTcuODk1MjVlKzA2 CTguNjMzMzRlKzA2Cgk2CTguMjM0OGUrMDYJNDgzMTY2CTcuNTI5NzVlKzA2CTguNzE3NDZlKzA2 Cgk2CTguMjY2OTFlKzA2CTM1NDI0OAk3Ljg1ODg3ZSswNgk4Ljg2NzRlKzA2Cgk2CTguMzc0M2Ur MDYJNTU4NDk5CTcuNzk0MDJlKzA2CTkuMzAyMzRlKzA2Cgk2CTguNzM3NDVlKzA2CTEzMTY5MAk4 LjUzOTQxZSswNgk4Ljg4NzgzZSswNgoJOAk2LjE3NjIzZSswNgk2OTIxNi4xCTYuMTIxNDllKzA2 CTYuMzM3ZSswNgoJOAk2LjIwMzNlKzA2CTE4Mzg5MAk1Ljk1MjllKzA2CTYuNTIwMzRlKzA2Cgk4 CTYuMjA1NWUrMDYJMTAzNTE1CTYuMDM2MDNlKzA2CTYuMzU3NzFlKzA2Cgk4CTYuMjU5NjllKzA2 CTE1MjgwMwk2LjA0ODM3ZSswNgk2LjQ4NDRlKzA2Cgk4CTYuMjYzNTFlKzA2CTcyMzM5LjIJNi4x NjExMmUrMDYJNi4zODI0MWUrMDYKCTEyCTQuMTE3NzllKzA2CTk4MDYyLjcJMy45NDIwMmUrMDYJ NC4yNDk1N2UrMDYKCTEyCTQuMTE5MjhlKzA2CTY0NzY4LjUJNC4wMDY2N2UrMDYJNC4yMzM5ZSsw NgoJMTIJNC4xNDM1MmUrMDYJNjU3NTcuNQk0LjAzNzAxZSswNgk0LjI0NDE1ZSswNgoJMTIJNC4x NjA1NGUrMDYJNjU3NTEuNgk0LjA0OTE3ZSswNgk0LjI3ODI1ZSswNgoJMTIJNC4xNzcwN2UrMDYJ NzA0NTkuNwk0LjAyOTkyZSswNgk0LjMwOTM2ZSswNgoJMTYJMy4wOTM4NmUrMDYJNTMyMjUuNQky Ljk4NDg4ZSswNgkzLjE4Mzg2ZSswNgoJMTYJMy4wOTkxN2UrMDYJNTQyMTUuMwkyLjk5MzM4ZSsw NgkzLjIwMDYyZSswNgoJMTYJMy4xMDI1OWUrMDYJNDkxNTYuOAkzLjAwMDllKzA2CTMuMjI1NmUr MDYKCTE2CTMuMTA0MzhlKzA2CTQ4ODcwLjkJMi45Nzg5N2UrMDYJMy4xODQ0MWUrMDYKCTE2CTMu MTA4NDhlKzA2CTUwOTA0LjMJMi45NDY3OWUrMDYJMy4xNzQ0OGUrMDYKCTMyCTEuNTUxNTNlKzA2 CTM3MzkzCTEuNDY0OGUrMDYJMS42MTM5ZSswNgoJMzIJMS41NTI2M2UrMDYJMzA2MTcuMwkxLjQ4 MzQ0ZSswNgkxLjYxMDQxZSswNgoJMzIJMS41NTU5M2UrMDYJMzAwNjEuMQkxLjQ4OTkxZSswNgkx LjYxMTQyZSswNgoJMzIJMS41NjM1NmUrMDYJMzY2ODcJMS41MTQ1M2UrMDYJMS42NDc4MWUrMDYK CTMyCTEuNTYzNTllKzA2CTM2NjExLjMJMS40ODM3MmUrMDYJMS42NTAyNGUrMDYKCTY0CTc2Nzcy MAkyMzc1My45CTcwODUyMQk4Mjc5MzgKCTY0CTc3MTIzNwkyOTU0MC41CTcwMTM2Mgk4NjMwODkK CTY0CTc3MjAwNwkzMDAzNi44CTcxNzA4MQk4NDk3NjkKCTY0CTc3NTc0MwkyNzEyNy45CTcwODUx Ngk4Mzk2ODkKCTY0CTc3NjA2OAkyMzAxOQk3Mzc0NzMJODQwMzY4CgkxMjgJMzgwODI0CTIxNTU1 CTMyODc1OAk0MzUwODQKCTEyOAkzODQ5MDIJMjE1OTYuMQkzMTI3NjIJNDI0NTQyCgkxMjgJMzg2 MDMxCTI0NTAxLjIJMzI4NzY3CTQ4MTQxOAoJMTI4CTM4Njg4NgkyMTA0NgkzMzg1NzgJNDQzODg4 CgkxMjgJMzg3NTgxCTIzNzcxCTM0NTUxMwk0NjY5ODMKCTI1NgkxOTA1MTEJMTUyMTYuNwkxNTMw NDUJMjI2OTIxCgkyNTYJMTkwOTY1CTE1NjcyLjEJMTQwMzI4CTI0NDI4NgoJMjU2CTE5MTU1Ngkx Njk3OAkxMzY3MTgJMjM3NzQ1CgkyNTYJMTkyMTA1CTE2NTA0LjIJMTU1NTE0CTIzOTc2MwoJMjU2 CTE5MjQwMwkxNTg5OC41CTE0NzExOQkyMzkwMjYK --=-uIGXMJ/Ke6Af2zSjv47N-- --=-3MtxOcsq7Me6T4zlxxYi Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEABECAAYFAlXjIdsACgkQD9PoadrVN+JeqgCglSWXlSEl8TfYkyacnEmlmNzO P1gAniPKgnGPEPGG1YXQfCALXL6zkzQ5 =8Pcx -----END PGP SIGNATURE----- --=-3MtxOcsq7Me6T4zlxxYi--