From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from krisdoz.my.domain (schwarze@localhost [127.0.0.1]) by krisdoz.my.domain (8.14.5/8.14.5) with ESMTP id s3BFkut5002048 for ; Fri, 11 Apr 2014 11:46:56 -0400 (EDT) Received: (from schwarze@localhost) by krisdoz.my.domain (8.14.5/8.14.3/Submit) id s3BFkqwi016835; Fri, 11 Apr 2014 11:46:52 -0400 (EDT) Date: Fri, 11 Apr 2014 11:46:52 -0400 (EDT) Message-Id: <201404111546.s3BFkqwi016835@krisdoz.my.domain> X-Mailinglist: mdocml-source Reply-To: source@mdocml.bsd.lv MIME-Version: 1.0 From: schwarze@mdocml.bsd.lv To: source@mdocml.bsd.lv Subject: mdocml: Further apropos(1) speed optimization was trickier than X-Mailer: activitymail 1.26, http://search.cpan.org/dist/activitymail/ Content-Type: text/plain; charset=utf-8 Log Message: ----------- Further apropos(1) speed optimization was trickier than anticipated. Contrary to what i initially thought, almost all time is now spent inside sqlite3(3) routines, and i found no easy way calling less of them. However, sqlite(3) spends substantial time in malloc(3), and even more (twice that) in its immediate malloc wrapper, sqlite3MemMalloc(), keeping track of all individual malloc chunk sizes. Typically about 90% of the malloced memory is used for purposes of the pagecache. By providing an mmap(3) MAP_ANON SQLITE_CONFIG_PAGECACHE, execution time decreases by 20-25% for simple (Nd and/or Nm) queries, 10-20% for non-NAME queries, and even apropos(1) resident memory size as reported by top(1) decreases by 20% for simple and by 60% for non-NAME queries. The new function, mansearch_setup(), spends no measurable time. The pagesize chosen is optimal: * Substantially smaller pages yield no gain at all. * Larger pages provide no additional benefit and just waste memory. The chosen number of pages in the cache is a compromise: * For simple queries, a handful of pages would suffice to get the full speed effect, at an apropos(1) resident memory size of about 2.0 MB. * For non-NAME queries, a large pagecache with 2k pages (2.5 MB) might gain a few more percent in speed, but at the expense of doubling the apropos(1) resident memory size for *all* queries. * The chosen number of 256 pages (330 kB) allows nearly full speed gain for all queries at the price of a 15% resident memory size increase. Modified Files: -------------- mdocml: apropos.c mansearch.c mansearch.h Revision Data ------------- Index: mansearch.h =================================================================== RCS file: /usr/vhosts/mdocml.bsd.lv/cvs/mdocml/mansearch.h,v retrieving revision 1.12 retrieving revision 1.13 diff -Lmansearch.h -Lmansearch.h -u -p -r1.12 -r1.13 --- mansearch.h +++ mansearch.h @@ -85,6 +85,7 @@ struct mansearch { #define MANSEARCH_WHATIS 0x01 /* whatis mode: equality, no key */ }; +int mansearch_setup(int); int mansearch(const struct mansearch *cfg, /* options */ const struct manpaths *paths, /* manpaths */ int argc, /* size of argv */ Index: mansearch.c =================================================================== RCS file: /usr/vhosts/mdocml.bsd.lv/cvs/mdocml/mansearch.c,v retrieving revision 1.27 retrieving revision 1.28 diff -Lmansearch.c -Lmansearch.c -u -p -r1.27 -r1.28 --- mansearch.c +++ mansearch.c @@ -19,6 +19,7 @@ #include "config.h" #endif +#include #include #include #include @@ -99,6 +100,53 @@ static void sql_match(sqlite3_context static void sql_regexp(sqlite3_context *context, int argc, sqlite3_value **argv); static char *sql_statement(const struct expr *); + +int +mansearch_setup(int start) +{ + static void *pagecache; + int c; + +#define PC_PAGESIZE 1280 +#define PC_NUMPAGES 256 + + if (start) { + if (NULL != pagecache) { + fprintf(stderr, "pagecache already enabled\n"); + return((int)MANDOCLEVEL_BADARG); + } + + pagecache = mmap(NULL, PC_PAGESIZE * PC_NUMPAGES, + PROT_READ | PROT_WRITE, MAP_ANON, -1, 0); + + if (MAP_FAILED == pagecache) { + perror("mmap"); + pagecache = NULL; + return((int)MANDOCLEVEL_SYSERR); + } + + c = sqlite3_config(SQLITE_CONFIG_PAGECACHE, + pagecache, PC_PAGESIZE, PC_NUMPAGES); + + if (SQLITE_OK == c) + return((int)MANDOCLEVEL_OK); + + fprintf(stderr, "pagecache: %s\n", sqlite3_errstr(c)); + + } else if (NULL == pagecache) { + fprintf(stderr, "pagecache missing\n"); + return((int)MANDOCLEVEL_BADARG); + } + + if (-1 == munmap(pagecache, PC_PAGESIZE * PC_NUMPAGES)) { + perror("munmap"); + pagecache = NULL; + return((int)MANDOCLEVEL_SYSERR); + } + + pagecache = NULL; + return((int)MANDOCLEVEL_OK); +} int mansearch(const struct mansearch *search, Index: apropos.c =================================================================== RCS file: /usr/vhosts/mdocml.bsd.lv/cvs/mdocml/apropos.c,v retrieving revision 1.37 retrieving revision 1.38 diff -Lapropos.c -Lapropos.c -u -p -r1.37 -r1.38 --- apropos.c +++ apropos.c @@ -95,6 +95,7 @@ main(int argc, char *argv[]) search.flags = whatis ? MANSEARCH_WHATIS : 0; manpath_parse(&paths, conf_file, defpaths, auxpaths); + mansearch_setup(1); ch = mansearch(&search, &paths, argc, argv, outkey, &res, &sz); manpath_free(&paths); @@ -110,6 +111,7 @@ main(int argc, char *argv[]) } free(res); + mansearch_setup(0); return(sz ? EXIT_SUCCESS : EXIT_FAILURE); usage: fprintf(stderr, "usage: %s [-C file] [-M path] [-m path] " -- To unsubscribe send an email to source+unsubscribe@mdocml.bsd.lv