ctags and ConTeXt

ctags and ConTeXt

[Aditya Mahajan]

Hi,

I am wondering how others manage a large list of reference labels in 
ConTeXt. I use Vim as an editor and it supports ctags. ctags is a 
utility that is basically for supporting tags in programs (initially 
only for C, and hence the name), but can be used relatively easily 
with LaTeX. Basically, ctags creates a tag file with the format

<tag-name> <tag-file> <tag-regex>

(for details see http://ctags.sourceforge.net/FORMAT)

Suppose I have a bunch of references all starting from eq:. Then 
inside Vim, I can type eq: and  press ^X^T for Vim to search for the 
tags file with everything starting with eq: and I get to see all of 
them in a pop up menu. I can choose which ever label I want and 
continue. Further I can press ^] on a tag, and Vim jumps to the 
location of the tag, something like a hyperlink within the editor. I 
find both these features very useful while working for files with more 
than 50 tags.

Creating a tag file for LaTeX is easy, since in LaTeX all labels are 
created using \label{....}. So, you can instruct ctags (the program) 
to search the file for \label{...} and store the first argument to the 
tags file.

However, taking this approach with ConTeXt is almost impossible. 
Since the label is tied to the environment that creates it, it is 
impossible to parse. There is no way a reg-ex can determine if mylabel 
in \startproblem[mylabel] is an optional argument or a label. So, the 
only way to really get the label is through TeX itself. The tui file 
has some information, but it can be very hard to parse. A typical tui 
file has enteries like

c 
\mainreference{}{cite-nsf-193}{2--0-2-4-2-0-0-0--9}{11}{MarschakRadner:1972}
c \mainreference{}{fnt:f:2}{2--0-2-4-1-0-0-0--8}{10}{}
c 
\mainreference{}{sec:prelim}{2--0-2-4-1-0-0-0--8}{10}{{C.4.1}{Preliminaries}}

where the first two are auto-generated, while I only want the user 
entered tags to be stored. Suppose, one is willing to have the extra 
overhead of parsing every mainreference in the tag file. I can get the 
first two pieces of information for the tag file, the name of the tag 
and the file that it is in. But how do I get a regex (or linenumber) 
of where the tag is located in the file, so that ^] takes me to that 
tag. I can not figure out a regex which will take care of every case.

Is there some other way to proceed? What are those other numbers in 
\mainreferece, do they contain any information about the line in the 
source file that generated the tag. And more generally, what do others 
do to keep track of labels in large projects.

Aditya

Re: ctags and ConTeXt

[Aditya Mahajan]

On Sun, 11 Feb 2007, Aditya Mahajan wrote:

> Hi,
>
> I am wondering how others manage a large list of reference labels in
> ConTeXt. I use Vim as an editor and it supports ctags. ctags is a
> utility that is basically for supporting tags in programs (initially
> only for C, and hence the name), but can be used relatively easily
> with LaTeX. Basically, ctags creates a tag file with the format
>
> <tag-name> <tag-file> <tag-regex>
>
> (for details see http://ctags.sourceforge.net/FORMAT)

I have a rudimentary support that is running. I have modified the 
writereference command so that it also writes the correct information 
to the tags file. It is almost working. I just need to figure out how 
to write a literal tab to a file? I tried

\immediate\write\file {tag^^I file^^I /:line_no/}

but this only writes a space. ctags expects \t to be present between 
tag and file name.

Secondly (this is not that important), can TeX write to a file without 
any extension? Traditionally the tags file is just named `tags'

Aditya

Re: ctags and ConTeXt

[Hans Hagen]

Aditya Mahajan wrote:
> but this only writes a space. ctags expects \t to be present between 
> tag and file name.
>   
try \rawcharacter{asciinumber}
> Secondly (this is not that important), can TeX write to a file without 
> any extension? Traditionally the tags file is just named `tags'
>   
did you try it? 

Hans 

-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                             | www.pragma-pod.nl
-----------------------------------------------------------------

Re: ctags and ConTeXt

[Aditya Mahajan]

On Mon, 26 Feb 2007, Hans Hagen wrote:

> Aditya Mahajan wrote:
>> but this only writes a space. ctags expects \t to be present between
>> tag and file name.
>>
> try \rawcharacter{asciinumber}

Thanks, \rawcharacter{9} works perfectly.

Aditya

Re: ctags and ConTeXt

[Taco Hoekwater]

Aditya Mahajan wrote:
> 
> Secondly (this is not that important), can TeX write to a file without 
> any extension? Traditionally the tags file is just named `tags'

No, it can't.

Taco

Re: ctags and ConTeXt

[Hans Hagen]

Taco Hoekwater wrote:
> Aditya Mahajan wrote:
>   
>> Secondly (this is not that important), can TeX write to a file without 
>> any extension? Traditionally the tags file is just named `tags'
>>     
>
> No, it can't.
>
>   
yet, in luatex we can overload the openers
> Taco
>
>
> _______________________________________________
> ntg-context mailing list
> ntg-context@ntg.nl
> http://www.ntg.nl/mailman/listinfo/ntg-context
>   


-- 

-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                             | www.pragma-pod.nl
-----------------------------------------------------------------

Re: ctags and ConTeXt

[Aditya Mahajan]

On Mon, 26 Feb 2007, Hans Hagen wrote:

> Taco Hoekwater wrote:
>> Aditya Mahajan wrote:
>>
>>> Secondly (this is not that important), can TeX write to a file without
>>> any extension? Traditionally the tags file is just named `tags'
>>>
>>
>> No, it can't.
>>
>>
> yet, in luatex we can overload the openers

For now, I will just write to a file like and tell vim to look for 
tags there. Once, I understand luatex a bit more, I will try to get a 
file without extension.

Aditya

Re: ctags and ConTeXt

[Aditya Mahajan]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 907 bytes --]

Hi

Here is my first attempt (see attachment). It works correctly in the 
little testing that I have done (on Windows). Just add 
\usemodule[ctags] somewhere in your environment file, and run context. 
I file called context.tags will be generated which contains the 
labels, filename, and line-number where the label occurs (line number 
can sometimes be off).

I modify a few core macros, so this can possibly break existing stuff 
(It should not, but then ...)

I found no reliable way to know the current file name. So this works 
correctly only if registerfileinfo is called. So one will have the 
correct filename when using a project structure, but not when using 
\input. I do not want to redefine \input, since almost everything 
relies on \input working correctly.


I will also post the module on the garden once I am more comfortable 
with it. This one is too dangerous to go to the core.

Aditya

[-- Attachment #2: Type: TEXT/PLAIN, Size: 5609 bytes --]

%M \logo [CTAGS] {CTAGS}

%D \module
%D   [       file=t-ctags,
%D        version=2007-02-26,
%D          title=\CONTEXT\ tags(\CTAGS),
%D       subtitle=\CTAGS\ support in \CONTEXT,
%D         author={Aditya Mahajan},
%D           date=\currentdate]
%D

%M \setuphead [section] [page=]
%M \useURL[ctags][http://ctags.sourceforge.net/]
%M \useURL[vim]  [http://www.vim.org]
%M \logo  [VIM] {VIM}
%M \logo  [EMACS] {EMACS}


%D \section Introduction
%D 
%D \CTAGS\ is a program that generates an index of language objects
%D found in a file that allows these objects to be easily located by a
%D text editor. They are tightly integrated with \VIM\ and \EMACS. For
%D large \TEX\ files, \CTAGS\ is very useful to find reference labels.
%D Suppose you are writing a long book or paper, and you want to add
%D reference to an equation. Also suppose all your equation labels start
%D with \type{eq:}. Then in \VIM\ you can type \type{eq:} and press
%D \type{^X^]} and \VIM\ will present you with a pop-up menu showing the
%D different labels that start with \type{eq:}. For long files this can
%D save you a lot of time.
%D
%D Traditionally tags are generated from an external program. However,
%D \CONTEXT's syntax is almost impossible to parse from an external
%D program. Consider the following two cases
%D \starttyping
%D \defineenumeration[problem]
%D \startproblem[tag]
%D blah blah
%D \stopproblem
%D \stoptyping
%D and
%D \starttyping
%D \defineitemgroup[problem]
%D \startproblem[intro]
%D \item blah blah
%D \stopproblem
%D \stoptyping
%D
%D For an external program to correctly parse this, it must be able to
%D understand \TEX. The only program that can understand \TEX\ is \TEX.
%D So, I decided to write \CTAGS\ support for \CONTEXT\ in \CONTEXT. It
%D turned out to be much simpler than expected. So now, one can read
%D \CTAGS\ as \CONTEXT\ tags \mono{:-)}.
%D
%D \section User Guide
%D
%D To use the module simply add 
%D \starttyping
%D \usemodule[ctags]
%D \stoptyping
%D on the top of the file\footnote{This must be done before
%D \type{\everystarttext} is executed. That is either in the environment
%D file or before \type{\starttext} in a file.}. Unfortunately, right
%D now (that is with \PDFTEX), it is not possible to create a file
%D without extension. So, the tags are written to a file
%D \filename{context.tags}. To use this in \VIM, you have to add
%D \starttyping
%D set tags=./tags,tags,context.tags
%D \stoptyping
%D in your \filename{.vimrc} (\filename{_vimrc} on Windows). I do not
%D know how to do something similar in \EMACS. If someone knows, please
%D let me know.
%D
%D This file modifies a core \CONTEXT\ macro, so be careful while using
%D it. It may (although it should not) break in some cases. You have
%D been warned.
%D
%D The rest of the file is self explanatory.


\writestatus  {loading}   {Context CTAGS support Module}

\startmodule[vim]

\unprotect

\newwrite\ctagsfile

\newtoks\everyopenctags
\newtoks\everyclosectags

%D TODO: Find out if we are running \LUATEX, and if we are write to
%D \filename{tags}.

\appendtoks
  \immediate\openout\ctagsfile context.tags
  \immediate\write\ctagsfile{!_TAG_FILE_FORMAT\rawcharacter{9}1}
  \immediate\write\ctagsfile{!_TAG_FILE_SORTED\rawcharacter{9}0}
  \immediate\write\ctagsfile{!_TAG_PROGRAM_AUTHOR\rawcharacter{9}Aditya Mahajan}
  \immediate\write\ctagsfile{!_TAG_PROGRAM_NAME\rawcharacter{9}ConTeXt t-ctags module}
\to \everyopenctags

\appendtoks
  \immediate\closeout\ctagsfile
\to \everyclosectags

\let\normalwriterefernce\writereference

\def\writereference#1% #2#3#4
  {\doifsomething{#1}
   {\immediate\write\ctagsfile{#1\rawcharacter{9}\currentctagsfile \rawcharacter{9}\the\inputlineno}}
   \normalwriterefernce{#1}}% #2#3#4

%D \type{\everystarttext} is executed at \type{\startcomponent}. If we
%D have a typical structure
%D \starttyping
%D \startcomponent file
%D \product  file
%D \enviromnent file
%D \stoptyping
%D then \type{\everystarttext} is executed before ctags module is
%D loaded. So, we open the tags as soon as the module is loaded. This
%D can cause a problem, if for whatever reason, \type{\everygoodbye} is
%D not executed.

\def\openctags{\the\everyopenctags}

\def\closectags{\the\everyclosectags}

% \appendtoks \openctags \to \everystarttext
\appendtoks \closectags \to \everygoodbye
% NOTE: Open the tags file!
\openctags


%D It does not seem possible to get the name of the current file being
%D processed. I tried \type{\inputfilename} and \type{\currentfilename}
%D but \type{\inputfilename} always gives the name of the product file,
%D while \type{\currentfilename} always get the name of the environment
%D file. So, I redefine \type{\registerfileinfo} to keep track of the
%D current file. This will only work if you have a proper project
%D structure, and not work with \type{\input}

\let\normalregisterfileinfo=\registerfileinfo

\def\registerfileinfo[#1#2]#3%
  {\doif{#1#2}{begin}
    {\splitctagsfilename{#3}
     \edef\currentctagsfile{\ctagsfilename.\ctagsfilesuffix}}
   \normalregisterfileinfo[#1#2]{#3}}

\def\splitctagsfilename#1
  {\edef\ascii{#1}\convertcommand\ascii\to\ascii%
   \splitstring\ascii\at.\to\ctagsfilename\and\ctagsfilesuffix%
   \lowercasestring\ctagsfilesuffix\to\ctafsfilesuffix%
   \doifnothing\ctagsfilename  {\let\ctagsfilename  \currentctagsfile}%
   \doifnothing\ctagsfilesuffix{\let\ctagsfilesuffix\c!tex}}
 
\def\currentctagsfile{\jobfullname}

\stopmodule
\protect

[-- Attachment #3: Type: text/plain, Size: 139 bytes --]

_______________________________________________
ntg-context mailing list
ntg-context@ntg.nl
http://www.ntg.nl/mailman/listinfo/ntg-context

Re: ctags and ConTeXt

[Hans Hagen]

Aditya Mahajan wrote:
> Hi
>
> Here is my first attempt (see attachment). It works correctly in the 
> little testing that I have done (on Windows). Just add 
> \usemodule[ctags] somewhere in your environment file, and run context. 
> I file called context.tags will be generated which contains the 
> labels, filename, and line-number where the label occurs (line number 
> can sometimes be off).
>
> I modify a few core macros, so this can possibly break existing stuff 
> (It should not, but then ...)
i didn't look at the code yet, but maybe a hook is a safer way?
>
> I found no reliable way to know the current file name. So this works 
> correctly only if registerfileinfo is called. So one will have the 
> correct filename when using a project structure, but not when using 
> \input. I do not want to redefine \input, since almost everything 
> relies on \input working correctly.
>
>
> I will also post the module on the garden once I am more comfortable 
> with it. This one is too dangerous to go to the core.
Hans 


-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                             | www.pragma-pod.nl
-----------------------------------------------------------------

Re: ctags and ConTeXt

[Aditya Mahajan]

On Tue, 27 Feb 2007, Hans Hagen wrote:

> Aditya Mahajan wrote:
>> Hi
>>
>> Here is my first attempt (see attachment). It works correctly in the
>> little testing that I have done (on Windows). Just add
>> \usemodule[ctags] somewhere in your environment file, and run context.
>> I file called context.tags will be generated which contains the
>> labels, filename, and line-number where the label occurs (line number
>> can sometimes be off).
>>
>> I modify a few core macros, so this can possibly break existing stuff
>> (It should not, but then ...)
> i didn't look at the code yet, but maybe a hook is a safer way?

The redefinition is safe. I put the warning since the user must be 
aware that a module redefines a core macro. There are two 
redefinitions, one for knowing the name of the current file, and the 
other for writings the reference to the tags file.

For knowing the name of the current file, I would say that the core 
should support it. From what I understand, \inputfilename (or 
\currentfile) should give the name of the current file, but they do 
not. So, I had to modify registerfileinfo so that it provides with the 
name of the current file.

To write the reference into the tag file, I need to know the 
reference. That is why I had to modify writereference. One hook that I 
can think of is to provide an \everywritereference token.

\def\writereference#1#2#3#4
   {\def\currentreferece{#1}
    \the\everywritereference
    % Rest of the writereference code}

Then, modules like ctags will be able to hook into 
everywritereference.

Aditya