Re: Improve performance of makewhatis

["Wesley Moore" <wes@wezm.net>]

Hi Ingo,

Thanks for the detailed reply.

On Wed, 15 Nov 2023, at 11:26 AM, Ingo Schwarze wrote:
>
> Can you confirm that these statements are accurate?
>
>  * Chimera Linux includes mandoc in its base system since 2021 Oct 30.

That's correct.

>  * There were no official nor unofficial packages of mandoc
>    for Chimera Linux before that.

Yes that's right.

>  * It uses the mandoc(1) parsing and formatting engine by default
>    since the same date.

Yes (pretty much). It looks like it was added a day later
https://github.com/chimera-linux/cports/commit/3c493733f7f0df345816cc69e273fca32fe70830

>  * The default apropos(1) program is mandoc since the same date.
>  * The default man(1) program is mandoc since the same date.
>  * Chimera Linux does not provide an officially supported
>    alternative for the man(1) program that users can select.
>  * Chimera Linux does not publish its manual pages on the web
>    on a site similar to https://manpages.debian.org/
>    or https://man.openbsd.org/ or https://man.voidlinux.org/ .

Yes all of above are correct.

>> makewhatis -Tutf8 is run to rebuild the index
>> whenever a package containing man pages is installed or updated.

>> I noticed this took multiple seconds even on a Ryzen 9 7950X system.
>
> That may not be a major problem because installing an additional
> package is not usually a fast operation (it usually requires both
> network access and expensive database locking and management outside
> the domain of manual pages) and it isn't done often either.
>
> So i expect most users won't feel offended if installing new
> software takes a few seconds.

Perhaps that is the case but it bothered me enough to go to the effort of writing these patches...

[snip]

> So only 10-15% of the time is spent in roff_getstrn(), which cannot
> possibly result in the 25-35% overall speedup you report, so something
> looks fishy here.  Then again, even 25-35% is not a large gain.

Those are the numbers I saw from my basic benchmarking measuring the wall clock time of makewhatis -Tutf -n across three runs and averaging the results for each machine. The raw data from my notes is available here:

https://gist.github.com/wezm/4043bde0dc2974c88b7706c60b58f900

[snip]

> One other thing.  I hate patch series, don't ever send them.
> They are usually a symptom of poor development methodology,
> and usually i reject them without even inspecting them.
> Here, i made an exception because your idea seems to have some merit.
>
> Every patch needs to
>  * perform one well-defined task and
>  * make the code better even if nothing else is ever committed
>    building on it.
>
> Splitting a patch like you did it here only adds obfuscation
> If a patch becomes too big for review, then the *task* it performs
> needs to be split into logically separate steps, rather than
> splitting code changes into mutiple patch files that essentially
> all save the same purpose and are similar and closely related in
> their content.  Such *logical* splitting can become tricky, but
> i doubt that is needed here.

Sorry about that. While I've been programming for a couple of decades the email based patch workflow is very new to me and different to all my experience to date. If I post any more changes I'll be sure to follow your suggestions.

> I'm not yet completely sure what the best way forward is.
> Probably something like:
>  1. Identifying all places that do string lookup in tables or lists.
>     Some can probably be left alone:
>       arch.c, att.c, cgi.c, chars.c, eqn.c, mansearch.c,
>       mdoc_argnames in mdoc.c, msec.c, st.c, regtab in roff.c,
>       tbl_layout.c, tbl_opts.c
>     But the others should probably be dealt with in a unified manner:
>       reqtab, mdocmac, manmac  on the one hand
>       strtab, rentab, pretab, xmbtab  on the other hand
>       Not sure yet whether xtab can be included into xmbtab.
>  2. Design and implement a common handling for these using ohash
>     that is as simple as possible.

When I have a moment I'll take a look through these and try to come up with an approach/plan and post it for feedback before making code changes.

> Not sure yet whether these are multiple steps or a single step,
> but doing one separate step for every *tab is definitely not the
> way.
>
> I guess that's enough for today...
>
> Yours,
>   Ingo

Wes
--
 To unsubscribe send an email to tech+unsubscribe@mandoc.bsd.lv