tech@mandoc.bsd.lv
 help / color / mirror / Atom feed
From: Ingo Schwarze <schwarze@usta.de>
To: Baptiste Daroussin <bapt@FreeBSD.org>
Cc: tech@mdocml.bsd.lv
Subject: Re: New db format
Date: Sat, 9 Jul 2016 18:12:51 +0200	[thread overview]
Message-ID: <20160709161251.GD6629@athene.usta.de> (raw)
In-Reply-To: <20160709152926.42bre6tittzmpm2g@ivaldir.etoilebsd.net>

Hi Baptiste,

Baptiste Daroussin FreeBSD wrote on Sat, Jul 09, 2016 at 05:29:27PM +0200:

> Beside minor portability issues (that I will provide fixes for once
> this patch is official)

You can already provide them now if you want to, and i'll include
them in my work-in-progress version and testing.

> I can say it works fine on FreeBSD :)
> Tested on FreeBSD 12-CURRENT:

Great to hear that.

My main worry is that i screwed alignment somewhere.  Did you test
on any machines with unusual alignment requirements?  Or is keeping
int32_t aligned to 4-byte boundaries guaranteed to be safe everywhere?

> The 3 main "bad" feedbacks I got about mandoc was:
> - time to create the db was long with sqlite with the new code
>   it is longer: does from 4s to 11s in general for the base manpages
>   but I do not consider that as a big issue given it is not run that
>   often and the code is quite new and there are probably room of
>   improvements

The SQLite version was optimized with gprof(1).  I can try again
for the new code.  But mandoc's makewhatis(8) will always remain
much slower than traditional makewhatis(8) because it parses the
complete manuals rather than just the NAME sections.

> - size of the db (for embedded). With the new code:
>   3.8M    /usr/share/man/mandoc.db
>   1.6M    /usr/share/man/mandoc.new.db
> 
>   This is a big improvement
> 
> - apropos can be very slow before showing anything, with the new code,
>   it is instant!
>   old 'apropos ls':
>           8.67 real         3.63 user         5.03 sys

That is surprising.  Yes, mandoc's SQLite based apropos(1) is slower
than the traditional BSD apropos(1) - no big surprise given the
much larger database.  But i have never seen it to be *that* slow
on OpenBSD, not even for complex queries accessing multiple macro
keys and using -a and -o; certainly not for a plain Nm,Nd access
like 'apropos ls' - that was always reasonably fast.  But now isn't
the time for making apropos-1.13 faster any longer...  :-)

schwarze@fantadrom $ dmesg
OpenBSD 5.8-stable (GENERIC) #1: Fri Oct 30 18:55:29 EST 2015
cpu0: Intel(R) Pentium(R) 4 CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
real mem  = 3203874816 (3055MB)
[...]
schwarze@fantadrom $ time apropos ls
[...]
    0m00.19s real     0m00.10s user     0m00.08s system

schwarze@man $ dmesg
OpenBSD 5.9 (GENERIC.MP) #1888: Fri Feb 26 01:20:19 MST 2016
cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3093.47 MHz
cpu1: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz
cpu2: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz
cpu3: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz
real mem = 8541536256 (8145MB)
[...]
schwarze@man $ time apropos ls
[...]
    0m00.02s real     0m00.01s user     0m00.01s system

schwarze@isnote $ dmesg
OpenBSD 6.0-beta (GENERIC.MP) #1875: Sun Jun 19 11:51:07 MDT 2016
cpu0: Genuine Intel(R) CPU T2300 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz
cpu1: Genuine Intel(R) CPU T2300 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz
real mem  = 3211083776 (3062MB)
[...]
schwarze@isnote $ time apropos ls
[...]
    0m00.22s real     0m00.09s user     0m00.07s system

>   new 'apropos ls':
>           0.02 real         0.02 user         0.00 sys
> 
> I notice some memory corruption in the ouput of apropos,
> I haven't dig into it yet

That might already be fixed; i just uploaded the current version
of the patch here:

  http://mdocml.bsd.lv/snapshots/mdocml-1.14.0.06.patch
  http://mdocml.bsd.lv/snapshots/mdocml-1.14.0.06.regress.tgz
  http://mdocml.bsd.lv/snapshots/mdocml-1.14.0.06.log.txt

THIS IS NOT A RELEASE AND NOT INTENDED FOR PRODUCTION.
This code is likely to be still buggy.

> Great work, I'm eager to see that code in!

Thanks for your support!

Yours,
  Ingo
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

  reply	other threads:[~2016-07-09 16:12 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <sfid-H20160709-151232-+015.17-1@spamfilter.osbf.lua>
     [not found] ` <20160709131158.GA6629@athene.usta.de>
2016-07-09 13:18   ` call for testing: mandoc-1.13.4 release candidate Thomas Klausner
2016-07-09 15:30     ` Ingo Schwarze
2016-07-09 13:39   ` Baptiste Daroussin FreeBSD
2016-07-09 14:31     ` Anthony J. Bentley
2016-07-09 15:29       ` New db format (was Re: call for testing: mandoc-1.13.4 release candidate) Baptiste Daroussin
2016-07-09 16:12         ` Ingo Schwarze [this message]
2016-08-05 20:26           ` New db format Baptiste Daroussin
2016-07-09 14:28   ` call for testing: mandoc-1.13.4 release candidate Svyatoslav Mishyn Crux
2016-07-10 14:08     ` Ingo Schwarze
2016-07-09 18:21   ` Svyatoslav Mishyn
2016-07-09 21:13     ` Ingo Schwarze
2016-07-09 18:55   ` Svyatoslav Mishyn Crux
2016-07-10 10:46     ` Svyatoslav Mishyn
2016-07-10 13:39       ` Ingo Schwarze
2016-07-10 14:27   ` Christian Neukirchen
2016-07-10 16:36     ` Ingo Schwarze
2016-07-10 16:48       ` Christian Neukirchen
2016-07-10 18:34         ` Ingo Schwarze

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160709161251.GD6629@athene.usta.de \
    --to=schwarze@usta.de \
    --cc=bapt@FreeBSD.org \
    --cc=tech@mdocml.bsd.lv \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).