Re: integrating context mkiv, luatex, and fmtutil, mktexlsr, etc

[Hans Hagen <pragma@wxs.nl>]

Norbert Preining wrote:
> On Mi, 03 Okt 2007, Hans Hagen wrote:
>> formats go to the paths defined in texmf.cnf (TEXFORMATS) and the 
>> filedatabses go alongside the ls-R files, so basically it then behaves 
>> like any other tex
> 
> ???  "file databases go alongside the ls-R files" what do you mean? I
> thought the stuff generated by luatools is something like the ls-R
> database for luatex. But it is placed NOT alongside the ls-R files,
> right?

well, "TEXMFSHARECACHE=yes" makes sure that it is put in the same path 
as the ls-r file

>> the cache path is only used and populated at runtime, so
> 
> So every time I run a luatex/context document it has to read ALL the
> trees again??? Cannot be, that was the whole point of ls-R/cache. It
> should be generated once (the cache).

no, not the trees, but if one uses a font, say lmroman10-regular.otf, 
then the 'converterd/prepared/whatever' datatable is cached so only that 
file ends up in the cache (flat structure, so, when one needs 
lmroman10-regular.otf, i first look in the cache (no database needed) 
for a tmc/a file, if not present i generate one, using the otf file from 
the normal tree, looked up in a kpse like way)

>> LUATEXCACHE=$TEXMFVAR/luacache:...
>> TEXMFSHARECACHE=yes
>>
>> should be ok then; if needed, later i can look into a way to share other 
>> cache stuff (say that you generate tmc files for 300 fonts at 
>> installation time) but if i understoof right, the main reason was 
>> formats and file databases
> 
> Yes, currently the main reason is for ls-R replacement and format
> location. But font data could be useful, too. My system has quite a lot
> of fonts ...

sure, but that demands some deep thinking because currently i use 
version numbers in the cached data and i don't want to open all cached 
files with the same name in order to check

> 
> 
> On Mi, 03 Okt 2007, Hans Hagen wrote:
>> texexec --make --luatex en
> [...]
>> this depends on the value of TEXFORMATS nd what path is first writable
> 
> Hmm,
> $ export TEXFORMATS=/home/norbert/.texmf-var/web2c
> $ texexec --make --luatex en
> ....
> TeXExec | using tex engine luatex
> TeXExec | using tex format path /home/norbert/.texmf-var/web2c/luatex
> TeXExec | generating tex format cont-en
> ....
> Transcript written on cont-en.log.

i know -) texexec only looks there and reports what it finds; texexec 
itself is unaware of lua and caches and ... i need to fix that

> LuaTools |
> LuaTools | runtime: 0.18 seconds
> TeXExec | no lua compilations needed
> TeXExec |
> TeXExec | tex engine path: /home/norbert/.texmf-var/web2c/luatex
> TeXExec |
> TeXExec |
> TeXExec | runtime: 9.486118
> $ ls ~/.texmf-var/web2c/luatex
> $ ls -l ~/luatex-cache/context/f7d1b3c25487ab1e1035aff1c53b90da/formats/
> -rw-r--r-- 1 norbert norbert 5669003 2007-10-03 14:07 cont-en.fmt
> -rw-r--r-- 1 norbert norbert   38786 2007-10-03 14:07 cont-en.log
> -rw-r--r-- 1 norbert norbert  159484 2007-10-03 14:07 cont-en.lua
> -rw-r--r-- 1 norbert norbert  112438 2007-10-03 14:07 cont-en.luc
> $
> 
> So the format is placed in some strange ;-) place under luatex-cache.

hm, also with the new version and the env var set? here it nicely goes 
to texmf-linux/web2c/luatex

btw, the 'strange' is just an md2 of the tree

> To sum it up: It is a bit unclear what purpose the cache is used for:
> - ls-R replacement, ie some sort of file database

indeed, more flexible, faster etc, and in the future we can use it for more

> - preprocessed font data cache so that loading the stuff is done faster

indeed, only at runtime, so normally in the users path someplace

> - formats??? what is saved for this

a cont-en.fmt as well as a cont-en.luc file, the format and it sstub

> But all of this is somehow static. On a normal system a normal user
> shouldn't have the necessity to change anything of the above. That
> should be done at install time of the respective stuff.

no, the font cache is not pregenerated at all, think of it like this:

- luatex loads otf font (takes time)
- provides it as table to mkiv (takes more time)
- mkiv adds a few things, preprocesses the font a bit
- then saves it as table (compiled) so that loading goes in an eyeblink

> Furthermore, the cache could be used (no idea whether this is true) for:
> - single job caching of data
> 	wouldn't it be better to keep generated files like this in the
> 	cwd, like .aux, etc files

we're talking of megabytes here

btw, context has no aux, toc etc file but a tuo file where all multi 
pass data goes into; in mkiv there is a lua companion because much of 
that data is not stored in lua tables (i still have to do the table of 
contents)

> What was the reason to do the hashing somewhere else but add the md5sums
> etc for these trees?

lengths of paths and such; i run luatex on the unix web/etc servers and 
have multiple trees in parallel, so now i can with one variable changed 
access all data;

/temp/luatex/context/<treeroot>/whatever


/temp/luatex/context/c:/blabla/tex/blabla/whatever


does not work that well, so i hash the treeroot

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------