temp file name conflicts in unattended document generation

temp file name conflicts in unattended document generation

[Ed L. Cashin]

Hi.  Most of the time, the files that are generated by TeX and its
friends during processing are named after the source file.  For
example, "foo.tex", when compiled, results in new files like foo.log,
foo.aux, foo.tui, etc., depending on the software in use.

I take advantage of that feature when making documents on behalf of
"client" processes, usually web servers.  Usually I'll add a process
number or a unique identifier for each potential client to the
filename, e.g., "foo-1234.tex".  

That way, the client's files don't step on the toes of other clients
who are trying to generate documents independently and maybe at the
same time.

I notice that occasionally I see a context-generated filename that is
not related to the source filename, e.g., "texutil.tuo" and
"cont-opt.bak".  It seems like these files may cause collisions when
two instances of context are running in the same directory on separate
sources, e.g. if, in /var/tmp, foo-2222.tex and foo-3333.tex are being
compiled simultaneously by context.

Is that a valid concern?  If so, is there a way to specify an "id"
string for use in the tempfile names?

-- 
--Ed Cashin                     PGP public key:
  ecashin@coe.uga.edu           http://www.coe.uga.edu/~ecashin/pgp/

Re: temp file name conflicts in unattended document generation

[Hans Hagen]

Ed L. Cashin wrote:

> I notice that occasionally I see a context-generated filename that is
> not related to the source filename, e.g., "texutil.tuo" and
> "cont-opt.bak".  It seems like these files may cause collisions when
> two instances of context are running in the same directory on separate
> sources, e.g. if, in /var/tmp, foo-2222.tex and foo-3333.tex are being
> compiled simultaneously by context.
> 
> Is that a valid concern?  If so, is there a way to specify an "id"
> string for use in the tempfile names?

I share your problem -) Which is why currently the opt file is already
taken care of. One reason for using a standard name, is that is eases
removing old files, (since I often run on overloaded systems this makes
sense). I think it indeed makes sense to treat the texutil.tuo file in
the same way, so I'll look into it. I have to updat emy perl 'purge'
script anyway -)

BTW, some file duplicated are hard to catch, like buffers with the ame
name and differrent content. But I can make that something optional,  

Remind me to solve this (I'm currently updating / reinstalling /
cleaning up my main system to a larger disk, so I don't play with the
sources currently).   

Hans 

-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.nl
-----------------------------------------------------------------

Re: temp file name conflicts in unattended document generation

[Taco Hoekwater]

>>>>> "Ed" == Ed L Cashin <ecashin@coe.uga.edu> writes:

    Ed> Is that a valid concern?  If so, is there a way to specify an
    Ed> "id" string for use in the tempfile names?

I agree with Ed here, there *is* a bit of a clash (also for
mpgraph.mp). But my solution is to use /var/tmp/$client-id as
directory, which solves the problem nicely.

Greetings, Taco

Re: temp file name conflicts in unattended document generation

[Wybo Dekker]

Hans Hagen wrote:
> I share your problem -) Which is why currently the opt file is already
> taken care of. One reason for using a standard name, is that is eases
> removing old files, (since I often run on overloaded systems this makes
> sense). I think it indeed makes sense to treat the texutil.tuo file in
> the same way, so I'll look into it. I have to updat emy perl 'purge'
> script anyway -)

Why don't you use $$.xxx for those files? You only need to remember to
remove $$.* at the end of you scripts.

-- 
Hartelijke groet, Wybo
--
__Servalys Analytical Chemistry Services__
Wybo H. Dekker      wybo@servalys.hobby.nl
Deilsedijk 60       www.hobby.nl/~servalys
4158 CH Deil        tel +31-345-652164    
The Netherlands     fax +31-345-652383    

Re: temp file name conflicts in unattended document generation

[Gilbert van den Dobbelsteen]

>
> Why don't you use $$.xxx for those files? You only need to remember to
> remove $$.* at the end of you scripts.

Nice trick, works fine in unix, but would probably break comptibiliy with
win23 platforms, unless the $$ is available.

For Hans: $$ returns the curent process Id in unix, which is guaranteed to
be unique. Of-course, Windows has process Id's but I don't know if you can
get to them from within perl. I wouldn't be surprised if that worked. Since
perl borrows much from shell-script programming, $$ is probably an existing
variable in perl which does the job.

Gilbert.

Re: temp file name conflicts in unattended document generation

[Taco Hoekwater]

    Gilbert> For Hans: $$ returns the curent process Id in unix, which
    Gilbert> is guaranteed to be unique. Of-course, Windows has
    Gilbert> process Id's but I don't know if you can get to them from
    Gilbert> within perl. I wouldn't be surprised if that
    Gilbert> worked. Since perl borrows much from shell-script
    Gilbert> programming, $$ is probably an existing variable in perl
    Gilbert> which does the job.

This works, at least in the newer perls (in the sense that you
actually get the process Id or at least something that is usable as
such). In the older perl ports, as well as in the DJGPP (DOS) port,
this will just give you a random number. Which should work just as
well regardless.

Greetings, Taco

Re: temp file name conflicts in unattended document generation

[Ed L. Cashin]

Taco Hoekwater <taco.hoekwater@wkap.nl> writes:

> >>>>> "Ed" == Ed L Cashin <ecashin@coe.uga.edu> writes:
> 
>     Ed> Is that a valid concern?  If so, is there a way to specify an
>     Ed> "id" string for use in the tempfile names?
> 
> I agree with Ed here, there *is* a bit of a clash (also for
> mpgraph.mp). But my solution is to use /var/tmp/$client-id as
> directory, which solves the problem nicely.

Wow, thanks, that's an excellent idea!  I heard about or saw someone
do that once and had forgotten the technique.

-- 
--Ed Cashin                     PGP public key:
  ecashin@coe.uga.edu           http://www.coe.uga.edu/~ecashin/pgp/

Re: temp file name conflicts in unattended document generation

[Ed L. Cashin]

Gilbert van den Dobbelsteen <gilbert@panter.soci.aau.dk> writes:

> For Hans: $$ returns the curent process Id in unix, which is
> guaranteed to be unique.

Disclaimer: although somewhat relevant to a discussion of unique
	    identifiers for document-generation runs, this is a bit
	    off topic.

Unique by process, yes, but an interesting development in perl is
mod-perl, where the compiled perl resides inside the web server
process until the web server dies.

That means that different invocations of the same perl code with the
same process id (viz., the web server's) could be producing different
documents.  So often I avoid the $$ technique for CGI and use a random
letter/number plus lockfiles instead.

I like Taco's suggestion of using a directory name that contains the
identifier for a given run. 

-- 
--Ed Cashin                     PGP public key:
  ecashin@coe.uga.edu           http://www.coe.uga.edu/~ecashin/pgp/

Re: temp file name conflicts in unattended document generation

[Hans Hagen]

Ed L. Cashin wrote:

> > I agree with Ed here, there *is* a bit of a clash (also for
> > mpgraph.mp). But my solution is to use /var/tmp/$client-id as
> > directory, which solves the problem nicely.
> 
> Wow, thanks, that's an excellent idea!  I heard about or saw someone
> do that once and had forgotten the technique.

In the current/next release when one says \protectbufferstrue in for
instance cont-sys.tex, temporary files will be prefixed. The protection
concerns the utility file copy, used to determine if more runs are
needed, the buffers used in a run and in figure inclusion (put a table
in a buffer and include it as 'external' figure and you can scale etc
like any figure), mpgraphics, mprun (a different kind of graphics), and
--still undocument but very well present-- the context_to_mp status info
file, which makes it possible to set up page backgrounds
spanning/combining areas, etc. Especially handling the mp files is
important, because in a next release I will introduce extensive embedded
graphic support, acting on graphic layers etc. Of course the texexec
option file is also protected now. 

Now, prefixing is handled low level by: 

\def\bufferprefix%
  {\ifprotectbuffers\jobname-\fi}

Technically, I can let texexec redefine this macro in the option file.
The best place for specifying the alternative prefix is probably the
texexec.ini file. Something: 

  for tetex set TeXScratchPath to /var/tmp/$client-id

or so. which would result in 

  print OPT "\\def\\bufferprefix{/var/tmp/$client-id/\n"

(of course I have to resolve the id first). 

This would at least provide a decent cleanup method. Now, how is this on
windows? 

Hans

-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.nl
-----------------------------------------------------------------

Re: temp file name conflicts in unattended document generation

[Gilbert van den Dobbelsteen]

> Gilbert van den Dobbelsteen <gilbert@panter.soci.aau.dk> writes:
>
> > For Hans: $$ returns the curent process Id in unix, which is
> > guaranteed to be unique.
>
> Disclaimer: although somewhat relevant to a discussion of unique
>     identifiers for document-generation runs, this is a bit
>     off topic.
>
> Unique by process, yes, but an interesting development in perl is
> mod-perl, where the compiled perl resides inside the web server
> process until the web server dies.

Right, that's what I'll be using in the future (when our new linux server
arrives). I will be using mod-perl for generating labels from forms input. I
am not sure if context is the path for our company, since what we need is
pretty trivial, and can easily be made using pdflib. On the other hand, I
can easily use plain pdftex, since this generates pdf files at high speed.

>
> That means that different invocations of the same perl code with the
> same process id (viz., the web server's) could be producing different
> documents.  So often I avoid the $$ technique for CGI and use a random
> letter/number plus lockfiles instead.
>
> I like Taco's suggestion of using a directory name that contains the
> identifier for a given run.

Taco's solution is very promising for the web-server approach. one can
easily obtain a unique client id, and run the stuff in different
directories, based on that id.

Gilbert.

Re: temp file name conflicts in unattended document generation

[Taco Hoekwater]

    Hans>   print OPT "\\def\\bufferprefix{/var/tmp/$client-id/\n"

    Hans> (of course I have to resolve the id first).

    Hans> This would at least provide a decent cleanup method. Now,
    Hans> how is this on windows?

dunno.

Taco

Re: temp file name conflicts in unattended document generation

[Ed L. Cashin]

Hans Hagen <pragma@wxs.nl> writes:

> In the current/next release when one says \protectbufferstrue in for
> instance cont-sys.tex, temporary files will be prefixed. The
> protection concerns the utility file copy, used to determine if more
> runs are needed, the buffers used in a run and in figure inclusion
> (put a table in a buffer and include it as 'external' figure and you
> can scale etc like any figure), mpgraphics, mprun (a different kind
> of graphics), and --still undocument but very well present-- the
> context_to_mp status info file, which makes it possible to set up
> page backgrounds spanning/combining areas, etc. Especially handling
> the mp files is important, because in a next release I will
> introduce extensive embedded graphic support, acting on graphic
> layers etc. Of course the texexec option file is also protected now.

I am really looking forward to learning to use those features.  I
think that some of them are documented in some presentation papers
by Hans Hagen that I read a long time ago.  

> Now, prefixing is handled low level by: 
> 
> \def\bufferprefix%
>   {\ifprotectbuffers\jobname-\fi}
> 
> Technically, I can let texexec redefine this macro in the option file.
> The best place for specifying the alternative prefix is probably the
> texexec.ini file. Something: 
> 
>   for tetex set TeXScratchPath to /var/tmp/$client-id
> 
> or so. which would result in 
> 
>   print OPT "\\def\\bufferprefix{/var/tmp/$client-id/\n"
> 
> (of course I have to resolve the id first). 
> 
> This would at least provide a decent cleanup method. Now, how is this on
> windows? 

It seems like any hard-coded value for a temp directory is bound to
cause problems.  

-- 
--Ed Cashin                     PGP public key:
  ecashin@coe.uga.edu           http://www.coe.uga.edu/~ecashin/pgp/

Re: temp file name conflicts in unattended document generation

[Hans Hagen]

Ed L. Cashin wrote:

> > the mp files is important, because in a next release I will
> > introduce extensive embedded graphic support, acting on graphic
> > layers etc. Of course the texexec option file is also protected now.
> 
> I am really looking forward to learning to use those features.  I
> think that some of them are documented in some presentation papers
> by Hans Hagen that I read a long time ago.

I suppose you refer to the presentation file in the NTS suite? That's
indeed what I'm working on. The interesting fact is that embedded
graphics introduce a sort of new way of thinking on graphics and text,
at least for me -), since one is dealing with the text flow as well as
layers, so what is covering what and depending on what and where and so
on. A main point is how to communicate between tex and mp and hide that
for the user. 

> >   for tetex set TeXScratchPath to /var/tmp/$client-id
> >
> > or so. which would result in
> >
> >   print OPT "\\def\\bufferprefix{/var/tmp/$client-id/\n"
> >
> ...  
> It seems like any hard-coded value for a temp directory is bound to
> cause problems.

Ok. Then I leave it as it is now and we'll see if this works out ok. 

Hans

-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.nl
-----------------------------------------------------------------

Re: temp file name conflicts in unattended document generation

[Ed L. Cashin]

Hans Hagen <pragma@wxs.nl> writes:

> Ed L. Cashin wrote:
> 
> > > the mp files is important, because in a next release I will
> > > introduce extensive embedded graphic support, acting on graphic
> > > layers etc. Of course the texexec option file is also protected now.
> > 
> > I am really looking forward to learning to use those features.  I
> > think that some of them are documented in some presentation papers
> > by Hans Hagen that I read a long time ago.
>
> I suppose you refer to the presentation file in the NTS suite?
> That's indeed what I'm working on. 

I don't know!  I was looking for them yesterday, but I've lost track
of where they are online.  They were presentations.  One paper would
do something interesting, usually with context and metapost, and then
the next one would go even further.  There were about five of them.

> The interesting fact is that embedded graphics introduce a sort of
> new way of thinking on graphics and text, at least for me -), since
> one is dealing with the text flow as well as layers, so what is
> covering what and depending on what and where and so on. A main
> point is how to communicate between tex and mp and hide that for the
> user.

That is an ambitious goal.  BTW, last night I gave a presentation on
perl programming to UGA's ACM, and many people there were very
impressed by the context-generated document that I was using for
handouts and a full-screen slide show.  I was able to get in some
proselytizing for context!

-- 
--Ed Cashin                     PGP public key:
  ecashin@coe.uga.edu           http://www.coe.uga.edu/~ecashin/pgp/

Re: temp file name conflicts in unattended document generation

[Hans Hagen]

Ed L. Cashin wrote:

> I don't know!  I was looking for them yesterday, but I've lost track
> of where they are online.  They were presentations.  One paper would
> do something interesting, usually with context and metapost, and then
> the next one would go even further.  There were about five of them.

On an undocumented temp path: 
  www.pragma-ade.nl/present/euronts/pre-nts.pdf

and pre-ntsa/pre-ntsb/pre-ntsc/pre-ntsd/pre-ntsx/pre-ntsy/pre-ntsz in
lowercase as you had already noticed. -)

Hans

-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.nl
-----------------------------------------------------------------

Re: temp file name conflicts in unattended document generation

[Hans Hagen]

Ed L. Cashin wrote:

> That is an ambitious goal.  BTW, last night I gave a presentation on
> perl programming to UGA's ACM, and many people there were very

I suppose you are aware of 

\startPL
  sub SomePerl 
    { $this = "nothing" }  
\stopPL 

which combined with \setupcolor[state=start] gives you a more fancy
output. 

(or: texexec --modu somefile.pl , given that you downloaded the
documentation zip). 

Hans

-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.nl
-----------------------------------------------------------------

Re: temp file name conflicts in unattended document generation

[Ed L. Cashin]

Hans Hagen <pragma@wxs.nl> writes:

> Ed L. Cashin wrote:
> 
> > That is an ambitious goal.  BTW, last night I gave a presentation on
> > perl programming to UGA's ACM, and many people there were very
> 
> I suppose you are aware of 
> 
> \startPL
>   sub SomePerl 
>     { $this = "nothing" }  
> \stopPL 
> 
> which combined with \setupcolor[state=start] gives you a more fancy
> output. 

No, I wasn't aware of that feature.  It's very interesting.  I was
using a method like this:

    \def\displayfile#1#2{%
            \startlinenumbering
            \typefile{#1}
            \stoplinenumbering\page}

(#2 was a short name, its use is not shown here) I very much like the
way it handles tabs.  I compared the two methods:

        http://www.coe.uga.edu/~ecashin/temp/perltest/

... and I noticed that the formatting looks nicer with \typefile
because of the way it handles tabs.  Tabs are kind of weird.  I should
look at the code for \typefile for educational purposes.

The green in the \startPL version is a bit hard to read, but I suppose
that's configurable.

-- 
--Ed Cashin                     PGP public key:
  ecashin@coe.uga.edu           http://www.coe.uga.edu/~ecashin/pgp/

Re: temp file name conflicts in unattended document generation

[Hans Hagen]

Ed L. Cashin wrote:

> ... and I noticed that the formatting looks nicer with \typefile
> because of the way it handles tabs.  Tabs are kind of weird.  I should
> look at the code for \typefile for educational purposes.
> 
> The green in the \startPL version is a bit hard to read, but I suppose
> that's configurable.

A few years ago, when writing articles for the MAPS, I found out that I
spent quite some time converting things to latex. I also needed some
features not present in latex. So, I rewrote the verbatim environment to
be generic. After that, taco started using the context verbatim macros
for the latex articles too (the maps editors accept both context and
latex input since there are two styles; actually, the maps was my main
reason for writing the gridsnapper, since I wanted the context articles
to have snapped column lines.) 

In the maps, taco uses different fonts instead of colors, so indeed, it
is configurable. BTW, when you look at the verb-* files as well as the
core-ver file, you will see that this is one place where color palets
are used. We use colors, because they match our editing environment
(that we wrote some 10 years ago and are converting to perl/tk now). 

There are several pretty printers: TEX, MP, PERL, JS, SQL, ..., and some
can be used mixed (which is what \newprettytrue enables). Actually, the
MP pretty printer handles for instance btex .. etex code as TEX, not as
MP. Of course there are limitations! It's just a substitute for real
literate programming, a thread that will be picked up soon.    

Another feature of the verbatim environment, is that display verbatim
will not produce one line at the bottom or top of a page. One can turn
off this feature. 

Hans

-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.nl
-----------------------------------------------------------------