From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.comp.tex.context/36855 Path: news.gmane.org!not-for-mail From: Hans Hagen Newsgroups: gmane.comp.tex.context.devel,gmane.comp.tex.context Subject: Re: integrating context mkiv, luatex, and fmtutil, mktexlsr, etc Date: Wed, 03 Oct 2007 15:06:43 +0200 Message-ID: <470393E3.1010803@wxs.nl> References: <20071003110634.GA29863@gamma.logic.tuwien.ac.at> <47037E8C.7080907@wxs.nl> <20071002135500.GH32335@gamma.logic.tuwien.ac.at> <47025A4D.6070404@wxs.nl> <20071003111513.GA31958@gamma.logic.tuwien.ac.at> <47037B3C.5030002@wxs.nl> <20071003074824.GW9324@gamma.logic.tuwien.ac.at> <47037739.208@wxs.nl> <20071003111038.GB29863@gamma.logic.tuwien.ac.at> <47037AF1.4020104@wxs.nl> <20071003122106.GA3597@gamma.logic.tuwien.ac.at> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: sea.gmane.org 1191416864 26931 80.91.229.12 (3 Oct 2007 13:07:44 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Wed, 3 Oct 2007 13:07:44 +0000 (UTC) Cc: dev-context@ntg.nl, ntg-context@ntg.nl To: Norbert Preining Original-X-From: dev-context-bounces@ntg.nl Wed Oct 03 15:07:37 2007 Return-path: Envelope-to: gctcd-dev-context@m.gmane.org Original-Received: from ronja.vet.uu.nl ([131.211.172.88] helo=ronja.ntg.nl) by lo.gmane.org with esmtp (Exim 4.50) id 1Id3wi-0007jv-Uq for gctcd-dev-context@m.gmane.org; Wed, 03 Oct 2007 15:07:25 +0200 Original-Received: from localhost (localhost [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id DEBB21FA3C; Wed, 3 Oct 2007 15:07:20 +0200 (CEST) Original-Received: from ronja.ntg.nl ([127.0.0.1]) by localhost (smtp.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 29441-05-4; Wed, 3 Oct 2007 15:07:11 +0200 (CEST) Original-Received: from ronja.vet.uu.nl (localhost [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id 7C32E1FB29; Wed, 3 Oct 2007 15:07:11 +0200 (CEST) Original-Received: from localhost (localhost [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id 411491FA9E; Wed, 3 Oct 2007 15:07:09 +0200 (CEST) Original-Received: from ronja.ntg.nl ([127.0.0.1]) by localhost (smtp.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 29441-05-3; Wed, 3 Oct 2007 15:07:04 +0200 (CEST) Original-Received: from mail.pragma-ade.net (dsl-083-247-100-017.solcon.nl [83.247.100.17]) by ronja.ntg.nl (Postfix) with ESMTP id 855221FB29; Wed, 3 Oct 2007 15:06:57 +0200 (CEST) Original-Received: from [10.100.1.100] (unverified [10.100.1.100]) by controller-1 (SurgeMail 3.7b8) with ESMTP id 17044 for multiple; Wed, 03 Oct 2007 15:06:51 +0200 User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) In-Reply-To: <20071003122106.GA3597@gamma.logic.tuwien.ac.at> X-Server: High Performance Mail Server - http://surgemail.com r=-274017400 X-Authenticated-User: hagen@controller-1 X-Virus-Scanned: amavisd-new at ntg.nl X-BeenThere: dev-context@ntg.nl X-Mailman-Version: 2.1.9 Precedence: list List-Id: "ConTeXt developers list. " List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: dev-context-bounces@ntg.nl Errors-To: dev-context-bounces@ntg.nl X-Virus-Scanned: amavisd-new at ntg.nl Xref: news.gmane.org gmane.comp.tex.context.devel:682 gmane.comp.tex.context:36855 Archived-At: Norbert Preining wrote: > On Mi, 03 Okt 2007, Hans Hagen wrote: >> formats go to the paths defined in texmf.cnf (TEXFORMATS) and the >> filedatabses go alongside the ls-R files, so basically it then behaves >> like any other tex > > ??? "file databases go alongside the ls-R files" what do you mean? I > thought the stuff generated by luatools is something like the ls-R > database for luatex. But it is placed NOT alongside the ls-R files, > right? well, "TEXMFSHARECACHE=yes" makes sure that it is put in the same path as the ls-r file >> the cache path is only used and populated at runtime, so > > So every time I run a luatex/context document it has to read ALL the > trees again??? Cannot be, that was the whole point of ls-R/cache. It > should be generated once (the cache). no, not the trees, but if one uses a font, say lmroman10-regular.otf, then the 'converterd/prepared/whatever' datatable is cached so only that file ends up in the cache (flat structure, so, when one needs lmroman10-regular.otf, i first look in the cache (no database needed) for a tmc/a file, if not present i generate one, using the otf file from the normal tree, looked up in a kpse like way) >> LUATEXCACHE=$TEXMFVAR/luacache:... >> TEXMFSHARECACHE=yes >> >> should be ok then; if needed, later i can look into a way to share other >> cache stuff (say that you generate tmc files for 300 fonts at >> installation time) but if i understoof right, the main reason was >> formats and file databases > > Yes, currently the main reason is for ls-R replacement and format > location. But font data could be useful, too. My system has quite a lot > of fonts ... sure, but that demands some deep thinking because currently i use version numbers in the cached data and i don't want to open all cached files with the same name in order to check > > > On Mi, 03 Okt 2007, Hans Hagen wrote: >> texexec --make --luatex en > [...] >> this depends on the value of TEXFORMATS nd what path is first writable > > Hmm, > $ export TEXFORMATS=/home/norbert/.texmf-var/web2c > $ texexec --make --luatex en > .... > TeXExec | using tex engine luatex > TeXExec | using tex format path /home/norbert/.texmf-var/web2c/luatex > TeXExec | generating tex format cont-en > .... > Transcript written on cont-en.log. i know -) texexec only looks there and reports what it finds; texexec itself is unaware of lua and caches and ... i need to fix that > LuaTools | > LuaTools | runtime: 0.18 seconds > TeXExec | no lua compilations needed > TeXExec | > TeXExec | tex engine path: /home/norbert/.texmf-var/web2c/luatex > TeXExec | > TeXExec | > TeXExec | runtime: 9.486118 > $ ls ~/.texmf-var/web2c/luatex > $ ls -l ~/luatex-cache/context/f7d1b3c25487ab1e1035aff1c53b90da/formats/ > -rw-r--r-- 1 norbert norbert 5669003 2007-10-03 14:07 cont-en.fmt > -rw-r--r-- 1 norbert norbert 38786 2007-10-03 14:07 cont-en.log > -rw-r--r-- 1 norbert norbert 159484 2007-10-03 14:07 cont-en.lua > -rw-r--r-- 1 norbert norbert 112438 2007-10-03 14:07 cont-en.luc > $ > > So the format is placed in some strange ;-) place under luatex-cache. hm, also with the new version and the env var set? here it nicely goes to texmf-linux/web2c/luatex btw, the 'strange' is just an md2 of the tree > To sum it up: It is a bit unclear what purpose the cache is used for: > - ls-R replacement, ie some sort of file database indeed, more flexible, faster etc, and in the future we can use it for more > - preprocessed font data cache so that loading the stuff is done faster indeed, only at runtime, so normally in the users path someplace > - formats??? what is saved for this a cont-en.fmt as well as a cont-en.luc file, the format and it sstub > But all of this is somehow static. On a normal system a normal user > shouldn't have the necessity to change anything of the above. That > should be done at install time of the respective stuff. no, the font cache is not pregenerated at all, think of it like this: - luatex loads otf font (takes time) - provides it as table to mkiv (takes more time) - mkiv adds a few things, preprocesses the font a bit - then saves it as table (compiled) so that loading goes in an eyeblink > Furthermore, the cache could be used (no idea whether this is true) for: > - single job caching of data > wouldn't it be better to keep generated files like this in the > cwd, like .aux, etc files we're talking of megabytes here btw, context has no aux, toc etc file but a tuo file where all multi pass data goes into; in mkiv there is a lua companion because much of that data is not stored in lua tables (i still have to do the table of contents) > What was the reason to do the hashing somewhere else but add the md5sums > etc for these trees? lengths of paths and such; i run luatex on the unix web/etc servers and have multiple trees in parallel, so now i can with one variable changed access all data; /temp/luatex/context//whatever /temp/luatex/context/c:/blabla/tex/blabla/whatever does not work that well, so i hash the treeroot Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------