Call for test documents

Call for test documents

[Norbert Preining]

Dear all,

I am currently testing the installation of ConTeXt/MkIV on Debian and
thus being in search for test documents using MkIV features/luatex
stuff. Or at least document which I can/should be able to process with
luatex/context.

If you have any documents it would be great if you could send me them,
best together with a short line on HOW to process the documents, and
even better with a "correct" pdf output.

This way I can do proper checking.

Thanks a lot and all the best

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining <preining@logic.at>        Vienna University of Technology
Debian Developer <preining@debian.org>                         Debian TeX Group
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
THRUMSTRER (n.)
The irritating man next to you in a concert who thinks he's (a) the
conductor, (b) the brass section.
			--- Douglas Adams, The Meaning of Liff

Re: [dev-context] Call for test documents

[Hans Hagen]

Norbert Preining wrote:
> Dear all,
> 
> I am currently testing the installation of ConTeXt/MkIV on Debian and
> thus being in search for test documents using MkIV features/luatex
> stuff. Or at least document which I can/should be able to process with
> luatex/context.
> 
> If you have any documents it would be great if you could send me them,
> best together with a short line on HOW to process the documents, and
> even better with a "correct" pdf output.

http://context.aanhet.net/svn/manuals/

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Call for test documents

[Norbert Preining]

On Di, 02 Okt 2007, Hans Hagen wrote:
> http://context.aanhet.net/svn/manuals/

Is there svn access? I have a checkout (very old) but the svn address
has changed.

Best wishes

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining <preining@logic.at>        Vienna University of Technology
Debian Developer <preining@debian.org>                         Debian TeX Group
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
SLUGGAN (n.)
A lurid facial bruise which everyone politely omits to mention because
it's obvious that you had a punch-up with your spouse last night - but
which into a door. It is useless to volunteer the true explanation
because nobody will believe it.
			--- Douglas Adams, The Meaning of Liff

Re: [dev-context] Call for test documents

[Hans Hagen]

Norbert Preining wrote:
> On Di, 02 Okt 2007, Hans Hagen wrote:
>> http://context.aanhet.net/svn/manuals/
> 
> Is there svn access? I have a checkout (very old) but the svn address
> has changed.

taco fetches from our internal svn server so he can give you the magic 
lines (sounds stupid, but since there's a firewall in between that 
redirects ports so any magic line i give is untested)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: [NTG-context] Call for test documents

[Taco Hoekwater]

Norbert Preining wrote:
> On Di, 02 Okt 2007, Hans Hagen wrote:
>> http://context.aanhet.net/svn/manuals/
> 
> Is there svn access? I have a checkout (very old) but the svn address
> has changed.

My mirror script (that populates the above url) uses:

    svn://83.247.100.17:33690/manuals

Re: integrating context mkiv, luatex, and fmtutil, mktexlsr, etc

[Norbert Preining]

On Mi, 03 Okt 2007, Hans Hagen wrote:
> formats go to the paths defined in texmf.cnf (TEXFORMATS) and the 
> filedatabses go alongside the ls-R files, so basically it then behaves 
> like any other tex

???  "file databases go alongside the ls-R files" what do you mean? I
thought the stuff generated by luatools is something like the ls-R
database for luatex. But it is placed NOT alongside the ls-R files,
right?

> the cache path is only used and populated at runtime, so

So every time I run a luatex/context document it has to read ALL the
trees again??? Cannot be, that was the whole point of ls-R/cache. It
should be generated once (the cache).

> LUATEXCACHE=$TEXMFVAR/luacache:...
> TEXMFSHARECACHE=yes
> 
> should be ok then; if needed, later i can look into a way to share other 
> cache stuff (say that you generate tmc files for 300 fonts at 
> installation time) but if i understoof right, the main reason was 
> formats and file databases

Yes, currently the main reason is for ls-R replacement and format
location. But font data could be useful, too. My system has quite a lot
of fonts ...



On Mi, 03 Okt 2007, Hans Hagen wrote:
> texexec --make --luatex en
[...]
> this depends on the value of TEXFORMATS nd what path is first writable

Hmm,
$ export TEXFORMATS=/home/norbert/.texmf-var/web2c
$ texexec --make --luatex en
...
TeXExec | using tex engine luatex
TeXExec | using tex format path /home/norbert/.texmf-var/web2c/luatex
TeXExec | generating tex format cont-en
...
Transcript written on cont-en.log.

LuaTools |
LuaTools | runtime: 0.18 seconds
TeXExec | no lua compilations needed
TeXExec |
TeXExec | tex engine path: /home/norbert/.texmf-var/web2c/luatex
TeXExec |
TeXExec |
TeXExec | runtime: 9.486118
$ ls ~/.texmf-var/web2c/luatex
$ ls -l ~/luatex-cache/context/f7d1b3c25487ab1e1035aff1c53b90da/formats/
-rw-r--r-- 1 norbert norbert 5669003 2007-10-03 14:07 cont-en.fmt
-rw-r--r-- 1 norbert norbert   38786 2007-10-03 14:07 cont-en.log
-rw-r--r-- 1 norbert norbert  159484 2007-10-03 14:07 cont-en.lua
-rw-r--r-- 1 norbert norbert  112438 2007-10-03 14:07 cont-en.luc
$

So the format is placed in some strange ;-) place under luatex-cache.

> btw, reminds me ... is this becoming a cross distribution linux
> convention (was already so on the mac) ... no longer root users, but
> always a sudu needed?

No, Debian does not install sudo at all per default. There is a real
root user.

-------------

To sum it up: It is a bit unclear what purpose the cache is used for:
- ls-R replacement, ie some sort of file database
- preprocessed font data cache so that loading the stuff is done faster
- formats??? what is saved for this

But all of this is somehow static. On a normal system a normal user
shouldn't have the necessity to change anything of the above. That
should be done at install time of the respective stuff.

Furthermore, the cache could be used (no idea whether this is true) for:
- single job caching of data
	wouldn't it be better to keep generated files like this in the
	cwd, like .aux, etc files
- caching of user files
	like the above, but initiated by the user

Again, I don't see any other use than these ones, but please enlighten
me what else it is used for. But in this case some mktexlsr-ish approach
would make sense.

So suggestion, like any TEXMF tree contains a ls-R file, we could have
for any TEXMFTREE a lcache/ directory $tree/lcache/ which contains the
cache for this tree.

Since luatools and other things extract the texmf trees anyway from the
TEXMFCNF variable, the cache could be read from there.

What was the reason to do the hashing somewhere else but add the md5sums
etc for these trees?


Best wishes

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining <preining@logic.at>        Vienna University of Technology
Debian Developer <preining@debian.org>                         Debian TeX Group
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
AINSWORTH (n.)
The length of time it takes to get served in a camera shop. Hence,
also, how long we will have to wait for the abolition of income tax or
the Second Coming.
			--- Douglas Adams, The Meaning of Liff

Re: integrating context mkiv, luatex, and fmtutil, mktexlsr, etc

[Hans Hagen]

Norbert Preining wrote:
> On Mi, 03 Okt 2007, Hans Hagen wrote:
>> formats go to the paths defined in texmf.cnf (TEXFORMATS) and the 
>> filedatabses go alongside the ls-R files, so basically it then behaves 
>> like any other tex
> 
> ???  "file databases go alongside the ls-R files" what do you mean? I
> thought the stuff generated by luatools is something like the ls-R
> database for luatex. But it is placed NOT alongside the ls-R files,
> right?

well, "TEXMFSHARECACHE=yes" makes sure that it is put in the same path 
as the ls-r file

>> the cache path is only used and populated at runtime, so
> 
> So every time I run a luatex/context document it has to read ALL the
> trees again??? Cannot be, that was the whole point of ls-R/cache. It
> should be generated once (the cache).

no, not the trees, but if one uses a font, say lmroman10-regular.otf, 
then the 'converterd/prepared/whatever' datatable is cached so only that 
file ends up in the cache (flat structure, so, when one needs 
lmroman10-regular.otf, i first look in the cache (no database needed) 
for a tmc/a file, if not present i generate one, using the otf file from 
the normal tree, looked up in a kpse like way)

>> LUATEXCACHE=$TEXMFVAR/luacache:...
>> TEXMFSHARECACHE=yes
>>
>> should be ok then; if needed, later i can look into a way to share other 
>> cache stuff (say that you generate tmc files for 300 fonts at 
>> installation time) but if i understoof right, the main reason was 
>> formats and file databases
> 
> Yes, currently the main reason is for ls-R replacement and format
> location. But font data could be useful, too. My system has quite a lot
> of fonts ...

sure, but that demands some deep thinking because currently i use 
version numbers in the cached data and i don't want to open all cached 
files with the same name in order to check

> 
> 
> On Mi, 03 Okt 2007, Hans Hagen wrote:
>> texexec --make --luatex en
> [...]
>> this depends on the value of TEXFORMATS nd what path is first writable
> 
> Hmm,
> $ export TEXFORMATS=/home/norbert/.texmf-var/web2c
> $ texexec --make --luatex en
> ....
> TeXExec | using tex engine luatex
> TeXExec | using tex format path /home/norbert/.texmf-var/web2c/luatex
> TeXExec | generating tex format cont-en
> ....
> Transcript written on cont-en.log.

i know -) texexec only looks there and reports what it finds; texexec 
itself is unaware of lua and caches and ... i need to fix that

> LuaTools |
> LuaTools | runtime: 0.18 seconds
> TeXExec | no lua compilations needed
> TeXExec |
> TeXExec | tex engine path: /home/norbert/.texmf-var/web2c/luatex
> TeXExec |
> TeXExec |
> TeXExec | runtime: 9.486118
> $ ls ~/.texmf-var/web2c/luatex
> $ ls -l ~/luatex-cache/context/f7d1b3c25487ab1e1035aff1c53b90da/formats/
> -rw-r--r-- 1 norbert norbert 5669003 2007-10-03 14:07 cont-en.fmt
> -rw-r--r-- 1 norbert norbert   38786 2007-10-03 14:07 cont-en.log
> -rw-r--r-- 1 norbert norbert  159484 2007-10-03 14:07 cont-en.lua
> -rw-r--r-- 1 norbert norbert  112438 2007-10-03 14:07 cont-en.luc
> $
> 
> So the format is placed in some strange ;-) place under luatex-cache.

hm, also with the new version and the env var set? here it nicely goes 
to texmf-linux/web2c/luatex

btw, the 'strange' is just an md2 of the tree

> To sum it up: It is a bit unclear what purpose the cache is used for:
> - ls-R replacement, ie some sort of file database

indeed, more flexible, faster etc, and in the future we can use it for more

> - preprocessed font data cache so that loading the stuff is done faster

indeed, only at runtime, so normally in the users path someplace

> - formats??? what is saved for this

a cont-en.fmt as well as a cont-en.luc file, the format and it sstub

> But all of this is somehow static. On a normal system a normal user
> shouldn't have the necessity to change anything of the above. That
> should be done at install time of the respective stuff.

no, the font cache is not pregenerated at all, think of it like this:

- luatex loads otf font (takes time)
- provides it as table to mkiv (takes more time)
- mkiv adds a few things, preprocesses the font a bit
- then saves it as table (compiled) so that loading goes in an eyeblink

> Furthermore, the cache could be used (no idea whether this is true) for:
> - single job caching of data
> 	wouldn't it be better to keep generated files like this in the
> 	cwd, like .aux, etc files

we're talking of megabytes here

btw, context has no aux, toc etc file but a tuo file where all multi 
pass data goes into; in mkiv there is a lua companion because much of 
that data is not stored in lua tables (i still have to do the table of 
contents)

> What was the reason to do the hashing somewhere else but add the md5sums
> etc for these trees?

lengths of paths and such; i run luatex on the unix web/etc servers and 
have multiple trees in parallel, so now i can with one variable changed 
access all data;

/temp/luatex/context/<treeroot>/whatever


/temp/luatex/context/c:/blabla/tex/blabla/whatever


does not work that well, so i hash the treeroot

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------

Re: [dev-context] integrating context mkiv, luatex, and fmtutil, mktexlsr, etc

[Norbert Preining]

HI Hans!

On Mi, 03 Okt 2007, Hans Hagen wrote:
> >database for luatex. But it is placed NOT alongside the ls-R files,
> >right?
> 
> well, "TEXMFSHARECACHE=yes" makes sure that it is put in the same path 
> as the ls-r file

Ah, ok, that is with the new context you created today. No, didn't test
this till now ;-) First I guess I need an updated luatex you said.

> no, not the trees, but if one uses a font, say lmroman10-regular.otf, 
> then the 'converterd/prepared/whatever' datatable is cached so only that 
> file ends up in the cache (flat structure, so, when one needs 
> lmroman10-regular.otf, i first look in the cache (no database needed) 
> for a tmc/a file, if not present i generate one, using the otf file from 
> the normal tree, looked up in a kpse like way)
[...]
> >- preprocessed font data cache so that loading the stuff is done faster
> 
> indeed, only at runtime, so normally in the users path someplace

Hmm, I still don't get it. In all this processing, is there a job
depending thing contained? I thought extracting all those tables etc can
be done independently of all jobs (and will help all jobs).

Anyway, I got it. So this is not a bad thing at all to have this stuff
generated at job running time, so we (integrators) don't have to worry
about it.

Only finding all the files is important for a start, so ls-R
replacement.


> >$ ls -l ~/luatex-cache/context/f7d1b3c25487ab1e1035aff1c53b90da/formats/
> >-rw-r--r-- 1 norbert norbert 5669003 2007-10-03 14:07 cont-en.fmt
> >-rw-r--r-- 1 norbert norbert   38786 2007-10-03 14:07 cont-en.log
> >-rw-r--r-- 1 norbert norbert  159484 2007-10-03 14:07 cont-en.lua
> >-rw-r--r-- 1 norbert norbert  112438 2007-10-03 14:07 cont-en.luc
> >$
> >
> >So the format is placed in some strange ;-) place under luatex-cache.
> 
> hm, also with the new version and the env var set? here it nicely goes 
> to texmf-linux/web2c/luatex

Probably an instance of "get the new beta", btw, where do I get it, from
the usual download place? Or is there something else for betas?

> >- formats??? what is saved for this
> 
> a cont-en.fmt as well as a cont-en.luc file, the format and it sstub

Yes of course, but what needs to be cached here but the sole information
that the file exist? (ie updating ls-R/cache)

> >But all of this is somehow static. On a normal system a normal user
> >shouldn't have the necessity to change anything of the above. That
> >should be done at install time of the respective stuff.
> 
> no, the font cache is not pregenerated at all, think of it like this:
> 
> - luatex loads otf font (takes time)
> - provides it as table to mkiv (takes more time)
> - mkiv adds a few things, preprocesses the font a bit
> - then saves it as table (compiled) so that loading goes in an eyeblink

Ok, so after the first processing later jobs will have access to these
tables in a blink. Ok, don't mind. Good for me ;-) No need to worry
about this.

> >Furthermore, the cache could be used (no idea whether this is true) for:
> >- single job caching of data
> >	wouldn't it be better to keep generated files like this in the
> >	cwd, like .aux, etc files
> 
> we're talking of megabytes here

Are these megabytes kept and are they useful for later runs? If yes,
then it is ok to put them in a separate cache (I assume the stuff is
useful, it is the font stuff).

> >What was the reason to do the hashing somewhere else but add the md5sums
> >etc for these trees?
> 
> lengths of paths and such; i run luatex on the unix web/etc servers and 
> have multiple trees in parallel, so now i can with one variable changed 
> access all data;
> 
> /temp/luatex/context/<treeroot>/whatever
> 
> 
> /temp/luatex/context/c:/blabla/tex/blabla/whatever
> 
> 
> does not work that well, so i hash the treeroot

Hmm, but if the caches would be in TEXMFTREE/lcache/ then the only thing
necessary would be a change of
	TEXMFCNF
referencing other trees.

(Sorry, only trying to understand the details ...)

Best wishes

Norbert

-------------------------------------------------------------------------------
Dr. Norbert Preining <preining@logic.at>        Vienna University of Technology
Debian Developer <preining@debian.org>                         Debian TeX Group
gpg DSA: 0x09C5B094      fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
-------------------------------------------------------------------------------
He stood up straight and looked the world squarely in the
fields and hills. To add weight to his words he stuck the
rabbit bone in his hair. He spread his arms out wide. `I
will go mad!' he announced.
                 --- Arthur discovering a way of coping with life on
                 --- Prehistoric Earth.
                 --- Douglas Adams, The Hitchhikers Guide to the Galaxy
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: [dev-context] integrating context mkiv, luatex, and fmtutil, mktexlsr, etc

[Hans Hagen]

Norbert Preining wrote:

> Are these megabytes kept and are they useful for later runs? If yes,
> then it is ok to put them in a separate cache (I assume the stuff is
> useful, it is the font stuff).

not sure yet, maybe later; but there is a cache cleaner (i may even 
thrash by date and usage)

> Hmm, but if the caches would be in TEXMFTREE/lcache/ then the only thing
> necessary would be a change of
> 	TEXMFCNF
> referencing other trees.
> 
> (Sorry, only trying to understand the details ...)

sure, all depends on what the cache var is set too (too many variants so 
i play safe and hash (but i can make that piece configurable if needed)

the beta is on the website and on the ftp

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________