From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk0-f174.google.com (mail-qk0-f174.google.com [209.85.220.174]) by fantadrom.bsd.lv (OpenSMTPD) with ESMTP id 666c7023 for ; Sat, 28 Jan 2017 08:02:49 -0500 (EST) Received: by mail-qk0-f174.google.com with SMTP id u25so88568931qki.2 for ; Sat, 28 Jan 2017 05:02:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=i3wm-org.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to; bh=j6RoWuQAZlg0xnCUSsLeJFkGzZgemsjy1XpcajKFNKI=; b=P8niGPPQVa7au0fe976DCawYCaBy3LiH0USaTdJAoWZSoAnN3G+uf6dv3PCzLsAJ0W Km/jPVQf3IbGxC0WRhhYnz4xRhhNuiB+7CKVh7v/pzDyqImrCLc7miYQ5cAxIKpLLssQ HVvsETgzoX6EEBxlO8NgUl+STE9IYqUWR8g900TXj8/6Yu7hWSMVnuPD2RrWCvo44CW4 e5NtjY8cQy1IKJ5iGRaTy6oBQuYW6SOp/hapt1wafxVXFy7sVKjSg3iOO5fBoe3wSPAI pdWMKjorVAxz+pSMS2znv8xMWjhDgyu/cxVY5rLuYIAHbvoLBUvAb8nyHuBk6P90fMrx 3MPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to; bh=j6RoWuQAZlg0xnCUSsLeJFkGzZgemsjy1XpcajKFNKI=; b=o3UADigimxbJp4GWjeFpAAlhlzm5scYuHEHsuaFr/kiOz+OvY9ysLiVhssQ6+WZhMY kqPvdNnI2QmASHG2dc59lvaWPSUqHfgm7ZwWdPs+N4gN/zdUFjL5D3iupPHdC7N87yDN cWo09jBwMn9iu66fGOfcfKXxQXmkXUFCICxcpI05zXn2ZcT7pinXON4OOgdXHHThyETg gagcq2PrFcUP9Uo3yIwwTbrjkUCKt098oJ9kmz1RSK8KK7H1OYYwmTIBLt1MLWgplbu8 fJstMSeFrcsqeFcl9vmdkcbJKVg2lp3ma6m6ZNMkr6TRuRdpaoCbubP7UgA/O03iFzhV K3FQ== X-Gm-Message-State: AIkVDXLXQrNAEsDT4m5WgIDFMTwW3DWTAGo4Jtt4ovjMIRLV4cxBZwSET1UIDM6otmbzX9zgMryjAoOJqEa6GQ== X-Received: by 10.55.164.77 with SMTP id n74mr2016369qke.322.1485608568992; Sat, 28 Jan 2017 05:02:48 -0800 (PST) X-Mailinglist: mdocml-tech Reply-To: tech@mdocml.bsd.lv MIME-Version: 1.0 Sender: michael@i3wm.org Received: by 10.55.98.83 with HTTP; Sat, 28 Jan 2017 05:02:28 -0800 (PST) X-Originating-IP: [2a02:168:4a00:0:225:90ff:fe5d:53a2] In-Reply-To: References: From: Michael Stapelberg Date: Sat, 28 Jan 2017 14:02:28 +0100 X-Google-Sender-Auth: 0-GX92eH6Dy9jCh99IlzlcciZuc Message-ID: Subject: Re: [PATCH] Implement -u (UNIX socket batch processing) To: tech@mdocml.bsd.lv Content-Type: multipart/mixed; boundary=94eb2c06bb2e07f32d05472733e9 --94eb2c06bb2e07f32d05472733e9 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Attaching revision 2 which resets ->gzip in mparse_reset(). I previously did not link against libz, so this issue escaped me. On Sat, Jan 21, 2017 at 1:29 PM, Michael Stapelberg wrote: > Quoting from the commit message for your convenience: > >> The option argument identifies the file descriptor number of a UNIX >> listening socket which was inherited from the parent process when said >> parent exec'd mandoc. >> >> mandoc will accept one connection at a time on that socket. As long as >> the connection lives, mandoc reads 1-byte messages with ancillary data >> (control information) from the connection. The ancillary data is used to >> pass three file descriptors from the parent process to mandoc. These >> file descriptors are then dup2'ed to fd 0, fd 1 and fd 2, respectively. >> >> After the file descriptors have been set up, a single manpage is read >> from stdin and processed to stdout/stderr. >> >> This change effectively adds a batch processing mode to mandoc, which >> eliminates the fork+exec overhead that otherwise accompanies the >> conversion of a large number of manpages. The resulting speed-up has >> been observed to exceed a factor of 6, reducing the total conversion >> wall-clock time of about a 470k manpages from more than 2 hours to >> a mere 22 minutes. > > To expand on the rationale: quicker conversion=E2=80=A6 > > 1. allows us to provide our users more recent manpages (we can run > more frequently) > 2. hogs fewer resources which can instead be used for actually serving ma= npages > 3. massively simplifies our disaster recovery story: we can just > re-generate the entire repository within a small number of hours, > instead of facing days of downtime > > As per http://man.openbsd.org/OpenBSD-current/man3/CMSG_DATA.3, > control information has been around since 4.2BSD, so support across > operating systems should be good. > > No file system objects are required, no data format needs to be > defined and no additional library dependencies are added. > > I have attached a minimal example implementation (demo.c) so that you > can convince yourself this approach works. > > Please let me know what you think. Thanks! > > -- > Best regards, > Michael --=20 Best regards, Michael --94eb2c06bb2e07f32d05472733e9 Content-Type: text/x-patch; charset=UTF-8; name="0001-Implement-u-UNIX-socket-batch-processing.patch" Content-Disposition: attachment; filename="0001-Implement-u-UNIX-socket-batch-processing.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_iyh8xxa12 RnJvbSA2OTA5YmY5NzU5ZTM3ZDIxZTQwMTZmZmViZWYxZTQ1M2EzYWJmYzA3IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBNaWNoYWVsIFN0YXBlbGJlcmcgPHN0YXBlbGJlcmdAZGViaWFu Lm9yZz4KRGF0ZTogVHVlLCAxNyBKYW4gMjAxNyAyMjoyNzoxNiArMDEwMApTdWJqZWN0OiBbUEFU Q0hdIEltcGxlbWVudCAtdSAoVU5JWCBzb2NrZXQgYmF0Y2ggcHJvY2Vzc2luZykKClRoZSBvcHRp b24gYXJndW1lbnQgaWRlbnRpZmllcyB0aGUgZmlsZSBkZXNjcmlwdG9yIG51bWJlciBvZiBhIFVO SVgKbGlzdGVuaW5nIHNvY2tldCB3aGljaCB3YXMgaW5oZXJpdGVkIGZyb20gdGhlIHBhcmVudCBw cm9jZXNzIHdoZW4gc2FpZApwYXJlbnQgZXhlYydkIG1hbmRvYy4KCm1hbmRvYyB3aWxsIGFjY2Vw dCBvbmUgY29ubmVjdGlvbiBhdCBhIHRpbWUgb24gdGhhdCBzb2NrZXQuIEFzIGxvbmcgYXMKdGhl IGNvbm5lY3Rpb24gbGl2ZXMsIG1hbmRvYyByZWFkcyAxLWJ5dGUgbWVzc2FnZXMgd2l0aCBhbmNp bGxhcnkgZGF0YQooY29udHJvbCBpbmZvcm1hdGlvbikgZnJvbSB0aGUgY29ubmVjdGlvbi4gVGhl IGFuY2lsbGFyeSBkYXRhIGlzIHVzZWQgdG8KcGFzcyB0aHJlZSBmaWxlIGRlc2NyaXB0b3JzIGZy b20gdGhlIHBhcmVudCBwcm9jZXNzIHRvIG1hbmRvYy4gVGhlc2UKZmlsZSBkZXNjcmlwdG9ycyBh cmUgdGhlbiBkdXAyJ2VkIHRvIGZkIDAsIGZkIDEgYW5kIGZkIDIsIHJlc3BlY3RpdmVseS4KCkFm dGVyIHRoZSBmaWxlIGRlc2NyaXB0b3JzIGhhdmUgYmVlbiBzZXQgdXAsIGEgc2luZ2xlIG1hbnBh Z2UgaXMgcmVhZApmcm9tIHN0ZGluIGFuZCBwcm9jZXNzZWQgdG8gc3Rkb3V0L3N0ZGVyci4KClRo aXMgY2hhbmdlIGVmZmVjdGl2ZWx5IGFkZHMgYSBiYXRjaCBwcm9jZXNzaW5nIG1vZGUgdG8gbWFu ZG9jLCB3aGljaAplbGltaW5hdGVzIHRoZSBmb3JrK2V4ZWMgb3ZlcmhlYWQgdGhhdCBvdGhlcndp c2UgYWNjb21wYW5pZXMgdGhlCmNvbnZlcnNpb24gb2YgYSBsYXJnZSBudW1iZXIgb2YgbWFucGFn ZXMuIFRoZSByZXN1bHRpbmcgc3BlZWQtdXAgaGFzCmJlZW4gb2JzZXJ2ZWQgdG8gZXhjZWVkIGEg ZmFjdG9yIG9mIDYsIHJlZHVjaW5nIHRoZSB0b3RhbCBjb252ZXJzaW9uCndhbGwtY2xvY2sgdGlt ZSBvZiBhYm91dCBhIDQ3MGsgbWFucGFnZXMgZnJvbSBtb3JlIHRoYW4gMiBob3VycyB0bwphIG1l cmUgMjIgbWludXRlcy4KLS0tCiBtYWluLmMgfCAxMTQgKysrKysrKysrKysrKysrKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKy0KIHJlYWQuYyB8ICAgMSAr CiAyIGZpbGVzIGNoYW5nZWQsIDExNCBpbnNlcnRpb25zKCspLCAxIGRlbGV0aW9uKC0pCgpkaWZm IC0tZ2l0IGEvbWFpbi5jIGIvbWFpbi5jCmluZGV4IGI2NGIzYmUuLjEyMjVhY2UgMTAwNjQ0Ci0t LSBhL21haW4uYworKysgYi9tYWluLmMKQEAgLTQxLDYgKzQxLDEwIEBACiAjaW5jbHVkZSA8dGlt ZS5oPgogI2luY2x1ZGUgPHVuaXN0ZC5oPgogCisjaW5jbHVkZSA8c3lzL3R5cGVzLmg+CisjaW5j bHVkZSA8c3lzL3NvY2tldC5oPgorI2luY2x1ZGUgPHN5cy91bi5oPgorCiAjaW5jbHVkZSAibWFu ZG9jX2F1eC5oIgogI2luY2x1ZGUgIm1hbmRvYy5oIgogI2luY2x1ZGUgInJvZmYuaCIKQEAgLTEw OSw2ICsxMTMsNTMgQEAgc3RhdGljCWNoYXIJCSAqaGVscF9hcmd2W10gPSB7aGVscF9hcmcsIE5V TEx9Owogc3RhdGljCWVudW0gbWFuZG9jbGV2ZWwgIHJjOwogCiAKKyNkZWZpbmUgTlVNX0ZEUyAz CitzdGF0aWMgaW50IHJlYWRfZmRzKGludCBjbGllbnRmZCwgaW50ICpmZHMpIHsKKwlzdHJ1Y3Qg bXNnaGRyCSAgbXNnOworCXVuc2lnbmVkIGNoYXIJICBkdW1teVsxXTsKKwlzdHJ1Y3QgaW92ZWMJ ICBpb3ZbMV07CisJc3NpemVfdAkJICBuOworCXN0cnVjdCBjbXNnaGRyCSAgKmNtc2c7CisJaW50 CQkgICp3YWxrOworCWludAkJICBjbnQ7CisJLyogVW5pb24gdXNlZCBmb3IgYWxpZ25tZW50LiAq LworCXVuaW9uIHsKKwkJdWludDhfdCBjb250cm9sYnVmW0NNU0dfU1BBQ0UoTlVNX0ZEUyAqIHNp emVvZihpbnQpKV07CisJCXN0cnVjdCBjbXNnaGRyIGFsaWduOworCX0gdTsKKworCW1lbXNldCgm bXNnLCAnXDAnLCBzaXplb2YobXNnKSk7CisJbXNnLm1zZ19jb250cm9sID0gdS5jb250cm9sYnVm OworCW1zZy5tc2dfY29udHJvbGxlbiA9IHNpemVvZih1LmNvbnRyb2xidWYpOworCS8qIFJlYWQg YSBkdW1teSBieXRlIOKAlCBzZW5kbXNnIGNhbm5vdCBzZW5kIGFuIGVtcHR5IG1lc3NhZ2UsCisJ ICogZXZlbiBpZiB3ZSBhcmUgb25seSBpbnRlcmVzdGVkIGluIHRoZSBPT0IgZGF0YS4gKi8KKwlp b3ZbMF0uaW92X2Jhc2UgPSBkdW1teTsKKwlpb3ZbMF0uaW92X2xlbiA9IHNpemVvZihkdW1teSk7 CisJbXNnLm1zZ19pb3YgPSBpb3Y7CisJbXNnLm1zZ19pb3ZsZW4gPSAxOworCisJaWYgKChuID0g cmVjdm1zZyhjbGllbnRmZCwgJm1zZywgMCkpIDwgMCkKKwkJcmV0dXJuIG47CisKKwlpZiAoKGNt c2cgPSBDTVNHX0ZJUlNUSERSKCZtc2cpKSA9PSBOVUxMKQorCQlyZXR1cm4gLTE7CisKKwlpZiAo Y21zZy0+Y21zZ19sZXZlbCAhPSBTT0xfU09DS0VUIHx8CisJICAgIGNtc2ctPmNtc2dfdHlwZSAh PSBTQ01fUklHSFRTKQorCQlyZXR1cm4gLTE7CisKKwlpZiAoY21zZy0+Y21zZ19sZW4gIT0gQ01T R19MRU4oTlVNX0ZEUyAqIHNpemVvZihpbnQpKSkKKwkJcmV0dXJuIC0xOworCisJd2FsayA9IChp bnQqKUNNU0dfREFUQShjbXNnKTsKKwlmb3IgKGNudCA9IDA7IGNudCA8IE5VTV9GRFM7IGNudCsr KSB7CisJCWZkc1tjbnRdID0gKndhbGs7CisJCXdhbGsrKzsKKwl9CisJcmV0dXJuIE5VTV9GRFM7 Cit9CisKKwogaW50CiBtYWluKGludCBhcmdjLCBjaGFyICphcmd2W10pCiB7CkBAIC0xMzMsNiAr MTg0LDE2IEBAIG1haW4oaW50IGFyZ2MsIGNoYXIgKmFyZ3ZbXSkKIAlpbnQJCSBzdGF0dXMsIHNp Z251bTsKIAlpbnQJCSBjOwogCXBpZF90CQkgcGFnZXJfcGlkLCB0Y19wZ2lkLCBtYW5fcGdpZCwg cGlkOworCWludAkJIHNvY2tmZDsKKwljb25zdCBjaGFyIAkgKmVycnN0cjsKKwlzdHJ1Y3Qgc29j a2FkZHJfdW4gcGVlcjsKKwlzb2NrbGVuX3QJIGxlbjsKKwlpbnQJCSBjbGllbnRmZDsKKwlpbnQJ CSBvbGRfc3RkaW47CisJaW50CQkgb2xkX3N0ZG91dDsKKwlpbnQJCSBvbGRfc3RkZXJyOworCWlu dAkJIGZkc1szXTsKKwlpbnQJCSBuOwogCiAjaWYgSEFWRV9QUk9HTkFNRQogCXByb2duYW1lID0g Z2V0cHJvZ25hbWUoKTsKQEAgLTE5NCw4ICsyNTUsMTEgQEAgbWFpbihpbnQgYXJnYywgY2hhciAq YXJndltdKQogCXNob3dfdXNhZ2UgPSAwOwogCW91dG1vZGUgPSBPVVRNT0RFX0RFRjsKIAorCXNv Y2tmZCA9IC0xOworCWVycnN0ciA9IE5VTEw7CisKIAl3aGlsZSAoLTEgIT0gKGMgPSBnZXRvcHQo YXJnYywgYXJndiwKLQkJCSJhQzpjZmhJOmlLOmtsTTptOk86UzpzOlQ6Vlc6dyIpKSkgeworCQkJ ImFDOmNmaEk6aUs6a2xNOm06TzpTOnM6VDp1OlZXOnciKSkpIHsKIAkJc3dpdGNoIChjKSB7CiAJ CWNhc2UgJ2EnOgogCQkJb3V0bW9kZSA9IE9VVE1PREVfQUxMOwpAQCAtMjYxLDYgKzMyNSwxMyBA QCBtYWluKGludCBhcmdjLCBjaGFyICphcmd2W10pCiAJCQlpZiAoICEgdG9wdGlvbnMoJmN1cnAs IG9wdGFyZykpCiAJCQkJcmV0dXJuIChpbnQpTUFORE9DTEVWRUxfQkFEQVJHOwogCQkJYnJlYWs7 CisJCWNhc2UgJ3UnOgorCQkJc29ja2ZkID0gc3RydG9udW0ob3B0YXJnLCAwLCBJTlRfTUFYLCAm ZXJyc3RyKTsKKwkJCWlmIChlcnJzdHIpIHsKKwkJCQl3YXJueCgiLXUgJXM6ICVzIiwgb3B0YXJn LCBlcnJzdHIpOworCQkJCXJldHVybiAoaW50KU1BTkRPQ0xFVkVMX0JBREFSRzsKKwkJCX0KKwkJ CWJyZWFrOwogCQljYXNlICdXJzoKIAkJCWlmICggISB3b3B0aW9ucygmY3VycCwgb3B0YXJnKSkK IAkJCQlyZXR1cm4gKGludClNQU5ET0NMRVZFTF9CQURBUkc7CkBAIC00MzYsNiArNTA3LDQ3IEBA IG1haW4oaW50IGFyZ2MsIGNoYXIgKmFyZ3ZbXSkKIAltY2hhcnNfYWxsb2MoKTsKIAljdXJwLm1w ID0gbXBhcnNlX2FsbG9jKG9wdGlvbnMsIGN1cnAud2xldmVsLCBtbXNnLCBkZWZvcyk7CiAKKwkv KiBJZiAtdSB3YXMgc3BlY2lmaWVkLCBhY2NlcHQgVU5JWCBzb2NrZXQgY29ubmVjdGlvbnMgdW50 aWwga2lsbGVkLiAqLworCXdoaWxlIChzb2NrZmQgPiAwKSB7CisJCWxlbiA9IHNpemVvZihzdHJ1 Y3Qgc29ja2FkZHJfdW4pOworCQljbGllbnRmZCA9IGFjY2VwdChzb2NrZmQsIChzdHJ1Y3Qgc29j a2FkZHIgKikmcGVlciwgJmxlbik7CisJCWlmIChjbGllbnRmZCA8IDApCisJCQllcnIoKGludClN QU5ET0NMRVZFTF9TWVNFUlIsICJhY2NlcHQiKTsKKworCQkvKiBXZSBhbHdheXMgc3dhcCBmaWxl IGRlc2NyaXB0b3JzIHNvIHRoYXQgd2UgY2FuIGFsd2F5cworCQkgKiB1bmNvbmRpdGlvbmFsbHkg dXNlIHRoZWlyIGZpbGUgZGVzY3JpcHRvciBudW1iZXJzLiAqLworCQlmZmx1c2goc3Rkb3V0KTsK KwkJZmZsdXNoKHN0ZGVycik7CisJCW9sZF9zdGRpbiA9IGR1cChTVERJTl9GSUxFTk8pOworCQlv bGRfc3Rkb3V0ID0gZHVwKFNURE9VVF9GSUxFTk8pOworCQlvbGRfc3RkZXJyID0gZHVwKFNUREVS Ul9GSUxFTk8pOworCisJCXdoaWxlICgxKSB7CisJCQlpZiAoKG4gPSByZWFkX2ZkcyhjbGllbnRm ZCwgZmRzKSkgPCAwKQorCQkJCWJyZWFrOworCisJCQlpZiAoZHVwMihmZHNbMF0sIFNURElOX0ZJ TEVOTykgPT0gLTEgfHwKKwkJCSAgICBkdXAyKGZkc1sxXSwgU1RET1VUX0ZJTEVOTykgPT0gLTEg fHwKKwkJCSAgICBkdXAyKGZkc1syXSwgU1RERVJSX0ZJTEVOTykgPT0gLTEpCisJCQkJZXJyKChp bnQpTUFORE9DTEVWRUxfU1lTRVJSLCAiZHVwMiIpOworCisJCQljbG9zZShmZHNbMF0pOworCQkJ Y2xvc2UoZmRzWzFdKTsKKwkJCWNsb3NlKGZkc1syXSk7CisKKwkJCXBhcnNlKCZjdXJwLCBTVERJ Tl9GSUxFTk8sICI8dW5peGZkPiIpOworCQkJbXBhcnNlX3Jlc2V0KGN1cnAubXApOworCQkJZmZs dXNoKHN0ZG91dCk7CisJCQlmZmx1c2goc3RkZXJyKTsKKwkJCS8qIENsb3NlIGZpbGUgZGVzY3Jp cHRvcnMgYnkgcmVzdG9yaW5nIHRoZSBvbGQgb25lcy4gKi8KKwkJCWR1cDIob2xkX3N0ZGluLCBT VERJTl9GSUxFTk8pOworCQkJZHVwMihvbGRfc3Rkb3V0LCBTVERPVVRfRklMRU5PKTsKKwkJCWR1 cDIob2xkX3N0ZGVyciwgU1RERVJSX0ZJTEVOTyk7CisJCX0KKworCQljbG9zZShjbGllbnRmZCk7 CisJfQorCiAJLyoKIAkgKiBDb25kaXRpb25hbGx5IHN0YXJ0IHVwIHRoZSBsb29rYXNpZGUgYnVm ZmVyIGJlZm9yZSBwYXJzaW5nLgogCSAqLwpkaWZmIC0tZ2l0IGEvcmVhZC5jIGIvcmVhZC5jCmlu ZGV4IGQyMGE2MDkuLmQ5NmViNWMgMTAwNjQ0Ci0tLSBhL3JlYWQuYworKysgYi9yZWFkLmMKQEAg LTgzNiw2ICs4MzYsNyBAQCBtcGFyc2VfcmVzZXQoc3RydWN0IG1wYXJzZSAqY3VycCkKIAogCWZy ZWUoY3VycC0+c29kZXN0KTsKIAljdXJwLT5zb2Rlc3QgPSBOVUxMOworCWN1cnAtPmd6aXAgPSAw OwogfQogCiB2b2lkCi0tIAoyLjExLjAKCg== --94eb2c06bb2e07f32d05472733e9-- -- To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv