From: Ingo Schwarze <schwarze@usta.de>
To: Baptiste Daroussin <bapt@FreeBSD.org>
Cc: tech@mdocml.bsd.lv
Subject: Re: New db format
Date: Sat, 9 Jul 2016 18:12:51 +0200 [thread overview]
Message-ID: <20160709161251.GD6629@athene.usta.de> (raw)
In-Reply-To: <20160709152926.42bre6tittzmpm2g@ivaldir.etoilebsd.net>
Hi Baptiste,
Baptiste Daroussin FreeBSD wrote on Sat, Jul 09, 2016 at 05:29:27PM +0200:
> Beside minor portability issues (that I will provide fixes for once
> this patch is official)
You can already provide them now if you want to, and i'll include
them in my work-in-progress version and testing.
> I can say it works fine on FreeBSD :)
> Tested on FreeBSD 12-CURRENT:
Great to hear that.
My main worry is that i screwed alignment somewhere. Did you test
on any machines with unusual alignment requirements? Or is keeping
int32_t aligned to 4-byte boundaries guaranteed to be safe everywhere?
> The 3 main "bad" feedbacks I got about mandoc was:
> - time to create the db was long with sqlite with the new code
> it is longer: does from 4s to 11s in general for the base manpages
> but I do not consider that as a big issue given it is not run that
> often and the code is quite new and there are probably room of
> improvements
The SQLite version was optimized with gprof(1). I can try again
for the new code. But mandoc's makewhatis(8) will always remain
much slower than traditional makewhatis(8) because it parses the
complete manuals rather than just the NAME sections.
> - size of the db (for embedded). With the new code:
> 3.8M /usr/share/man/mandoc.db
> 1.6M /usr/share/man/mandoc.new.db
>
> This is a big improvement
>
> - apropos can be very slow before showing anything, with the new code,
> it is instant!
> old 'apropos ls':
> 8.67 real 3.63 user 5.03 sys
That is surprising. Yes, mandoc's SQLite based apropos(1) is slower
than the traditional BSD apropos(1) - no big surprise given the
much larger database. But i have never seen it to be *that* slow
on OpenBSD, not even for complex queries accessing multiple macro
keys and using -a and -o; certainly not for a plain Nm,Nd access
like 'apropos ls' - that was always reasonably fast. But now isn't
the time for making apropos-1.13 faster any longer... :-)
schwarze@fantadrom $ dmesg
OpenBSD 5.8-stable (GENERIC) #1: Fri Oct 30 18:55:29 EST 2015
cpu0: Intel(R) Pentium(R) 4 CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
real mem = 3203874816 (3055MB)
[...]
schwarze@fantadrom $ time apropos ls
[...]
0m00.19s real 0m00.10s user 0m00.08s system
schwarze@man $ dmesg
OpenBSD 5.9 (GENERIC.MP) #1888: Fri Feb 26 01:20:19 MST 2016
cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3093.47 MHz
cpu1: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz
cpu2: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz
cpu3: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz
real mem = 8541536256 (8145MB)
[...]
schwarze@man $ time apropos ls
[...]
0m00.02s real 0m00.01s user 0m00.01s system
schwarze@isnote $ dmesg
OpenBSD 6.0-beta (GENERIC.MP) #1875: Sun Jun 19 11:51:07 MDT 2016
cpu0: Genuine Intel(R) CPU T2300 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz
cpu1: Genuine Intel(R) CPU T2300 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz
real mem = 3211083776 (3062MB)
[...]
schwarze@isnote $ time apropos ls
[...]
0m00.22s real 0m00.09s user 0m00.07s system
> new 'apropos ls':
> 0.02 real 0.02 user 0.00 sys
>
> I notice some memory corruption in the ouput of apropos,
> I haven't dig into it yet
That might already be fixed; i just uploaded the current version
of the patch here:
http://mdocml.bsd.lv/snapshots/mdocml-1.14.0.06.patch
http://mdocml.bsd.lv/snapshots/mdocml-1.14.0.06.regress.tgz
http://mdocml.bsd.lv/snapshots/mdocml-1.14.0.06.log.txt
THIS IS NOT A RELEASE AND NOT INTENDED FOR PRODUCTION.
This code is likely to be still buggy.
> Great work, I'm eager to see that code in!
Thanks for your support!
Yours,
Ingo
--
To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv
next prev parent reply other threads:[~2016-07-09 16:12 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <sfid-H20160709-151232-+015.17-1@spamfilter.osbf.lua>
[not found] ` <20160709131158.GA6629@athene.usta.de>
2016-07-09 13:18 ` call for testing: mandoc-1.13.4 release candidate Thomas Klausner
2016-07-09 15:30 ` Ingo Schwarze
2016-07-09 13:39 ` Baptiste Daroussin FreeBSD
2016-07-09 14:31 ` Anthony J. Bentley
2016-07-09 15:29 ` New db format (was Re: call for testing: mandoc-1.13.4 release candidate) Baptiste Daroussin
2016-07-09 16:12 ` Ingo Schwarze [this message]
2016-08-05 20:26 ` New db format Baptiste Daroussin
2016-07-09 14:28 ` call for testing: mandoc-1.13.4 release candidate Svyatoslav Mishyn Crux
2016-07-10 14:08 ` Ingo Schwarze
2016-07-09 18:21 ` Svyatoslav Mishyn
2016-07-09 21:13 ` Ingo Schwarze
2016-07-09 18:55 ` Svyatoslav Mishyn Crux
2016-07-10 10:46 ` Svyatoslav Mishyn
2016-07-10 13:39 ` Ingo Schwarze
2016-07-10 14:27 ` Christian Neukirchen
2016-07-10 16:36 ` Ingo Schwarze
2016-07-10 16:48 ` Christian Neukirchen
2016-07-10 18:34 ` Ingo Schwarze
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160709161251.GD6629@athene.usta.de \
--to=schwarze@usta.de \
--cc=bapt@FreeBSD.org \
--cc=tech@mdocml.bsd.lv \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).