An idea

An idea

[Zeljko Vrba]

A motivation and idea that came to mind while writing a post to comp.text.tex:

==
Then again, most (all?) of publishers require Latex source, if my document
ever makes it that far.. And Latex has nicer source code formatting than
Context. so I'm really split between Latex and Context..

None of them individually fit all (I admit, vague) requirements.. Context
being able to chew up Latex .cls files and still allowing for its advanced
formatting directives, long tables, and other goodies would be a perfect
match.
==

Basically, rewrite standard Latex environments and commands (\chapter,
\section, itemize, enumerate, etc..) present in article.cls and report.cls
to expand to Context commands that give an equivalent layout..

How much work would it be? Either a Latex emulation environment or a new
implementation of "Latex classes" for Context?

I have to admit that I don't have time to start implementing either. I'm just
interested in some feasibility opinions..

80% of Latex<>Context transition can be done with a few simple regexes.
Other 20% have to be translated by hand, and it is very painful, esp.
if tables are in question..

Re: An idea

[Hans Hagen]

Zeljko Vrba wrote:

>A motivation and idea that came to mind while writing a post to comp.text.tex:
>
>==
>Then again, most (all?) of publishers require Latex source, if my document
>ever makes it that far.. And Latex has nicer source code formatting than
>Context. so I'm really split between Latex and Context..
>
>  
>
what do you mean with source code formatting?

>None of them individually fit all (I admit, vague) requirements.. Context
>being able to chew up Latex .cls files and still allowing for its advanced
>formatting directives, long tables, and other goodies would be a perfect
>match.
>==
>
>Basically, rewrite standard Latex environments and commands (\chapter,
>\section, itemize, enumerate, etc..) present in article.cls and report.cls
>to expand to Context commands that give an equivalent layout..
>
>How much work would it be? Either a Latex emulation environment or a new
>implementation of "Latex classes" for Context?
>
>I have to admit that I don't have time to start implementing either. I'm just
>interested in some feasibility opinions..
>
>80% of Latex<>Context transition can be done with a few simple regexes.
>Other 20% have to be translated by hand, and it is very painful, esp.
>if tables are in question..
>  
>
brooks is working on that so maybe you should team up with him

Hans

Re: An idea

[Nikolai Weibull]

Zeljko Vrba wrote:

> A motivation and idea that came to mind while writing a post to
> comp.text.tex:
> 
> ==
> Then again, most (all?) of publishers require Latex source, if my
> document ever makes it that far.. And Latex has nicer source code
> formatting than Context. so I'm really split between Latex and
> Context..
> 
> None of them individually fit all (I admit, vague) requirements..
> Context being able to chew up Latex .cls files and still allowing for
> its advanced formatting directives, long tables, and other goodies
> would be a perfect match.
> ==

We were planning on using Vim for preprocessing things like this instead
of hacking TeX to do the lexing.  I think Mojca (?) had some ideas for
this and perhaps even a simple implementation, and given that there is
now structured support for preprocessing in texexec I think that this
can be done quite easily.  A lot easier than writing something in TeX.

        nikolai

-- 
Nikolai Weibull: now available free of charge at http://bitwi.se/!
Born in Chicago, IL USA; currently residing in Gothenburg, Sweden.
main(){printf(&linux["\021%six\012\0"],(linux)["have"]+"fun"-97);}

Re: An idea

[Taco Hoekwater]

Nikolai Weibull wrote:
> We were planning on using Vim for preprocessing things like this instead
> of hacking TeX to do the lexing.  I think Mojca (?) had some ideas for
> this and perhaps even a simple implementation, and given that there is
> now structured support for preprocessing in texexec I think that this
> can be done quite easily.  A lot easier than writing something in TeX.

Speak for yourself. I'll take TeX over VIM anytime :-)

Taco

Re: An idea

[Zeljko Vrba]

On Fri, Dec 02, 2005 at 03:14:30PM +0100, Hans Hagen wrote:
>
> what do you mean with source code formatting?
> 
Something akin to lgrind. Pretty-printing C++ source code. I tried to
adapt some existing Context code to C and C++ keywords, but failed
somewhere.. AFAIR, I may have even once asked on this list about problems
about variable declarations like &a where the & got swallowed up..

Anyway, I'm happy with the default typing environments, so that's one of the
reasons I've given up on making it work. There isn't much use of color in
black-white printing anyway. Grayscales just reduce the contrast and make the
text harder to read.

>
> brooks is working on that so maybe you should team up with him
> 
hm! maybe I could find a little spare time to help him. i'm really annoyed
by many of Latex's misfeatures...

Best regards,
  Zeljko.

Re: An idea

[Zeljko Vrba]

On Fri, Dec 02, 2005 at 05:56:36PM +0100, Nikolai Weibull wrote:
> 
> We were planning on using Vim for preprocessing things like this instead
>
Vim for preprocessing? Isn't it a bit too painful compared to using a "real"
programming language?

>
> of hacking TeX to do the lexing.  I think Mojca (?) had some ideas for
>
I would agree with that. I never did any serious programming in TeX apart
from few simple \defs.

>
> this and perhaps even a simple implementation, and given that there is
> now structured support for preprocessing in texexec I think that this
> can be done quite easily.  A lot easier than writing something in TeX.
> 
any links to the project?

Re: An idea

[Nikolai Weibull]

Zeljko Vrba wrote:

> On Fri, Dec 02, 2005 at 05:56:36PM +0100, Nikolai Weibull wrote:

> > We were planning on using Vim for preprocessing things like this
> > instead

> Vim for preprocessing? Isn't it a bit too painful compared to using a
> "real" programming language?

You misunderstand.  Vim can easily be made to output the highlighting it
would apply to a document on the display to something else, e.g.,
outputting TeX directives instead.

A subthread on the subject can be found here:

http://www.ntg.nl/pipermail/ntg-context/2005/012885.html

It would require very little programming.  syntax/2html.vim, which
converts the buffer to a HTML document with syntax highlighting, is 526
lines in the current CVS incarnation.  A syntax/2context.vim would be
even shorter, perhaps 150 to 200 lines.  If I find the time I’ll write
something this weekend.  I’m catching a could though, so I might not
:-(.

(Note: This would thus allow highlighted code for any format Vim
understand, which is currently over 450!)

        nikolai

-- 
Nikolai Weibull: now available free of charge at http://bitwi.se/!
Born in Chicago, IL USA; currently residing in Gothenburg, Sweden.
main(){printf(&linux["\021%six\012\0"],(linux)["have"]+"fun"-97);}

Re[2]: An idea

[Giuseppe Bilotta]

Friday, December 2, 2005 Taco Hoekwater wrote:



> Nikolai Weibull wrote:
>> We were planning on using Vim for preprocessing things like this instead
>> of hacking TeX to do the lexing.  I think Mojca (?) had some ideas for
>> this and perhaps even a simple implementation, and given that there is
>> now structured support for preprocessing in texexec I think that this
>> can be done quite easily.  A lot easier than writing something in TeX.

> Speak for yourself. I'll take TeX over VIM anytime :-)

Eh, I'll have both TeX and Vim anytime :)

Sheesh, even my IRC identity uses tex_vim as username and
the UTF-8 equivalent of

$\tau\epsilon\chi\nu\epsilon$ \&\ vis

as "Real name" :)

-- 
Giuseppe "Oblomov" Bilotta

Re: An idea

[Nikolai Weibull]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

Nikolai Weibull wrote:

> It would require very little programming.  syntax/2html.vim, which
> converts the buffer to a HTML document with syntax highlighting, is 526
> lines in the current CVS incarnation.  A syntax/2context.vim would be
> even shorter, perhaps 150 to 200 lines.  If I find the time I’ll write
> something this weekend.  I’m catching a could though, so I might not
> :-(.

A splitting headache notwithstanding, here’s a syntax/2context.vim that
weighs in at 170 lines.  There are still things to do, like figuring out
how to complement this on the ConTeXt side (i.e., defining \highlight)
and things will depend on how this is done.  Some sort of \type
environment would be nice, as it is better to not do escaping of special
characters on the Vim side.  Someone with better knowledge of how to do
this than I have is welcome to finish it.  The \highlight command should
be defined something like this (pseudo-tex-code):

\pdef\highlight[#1]{#2}%
  {\bgroup
   \setupcolorforgroup[#1]%
   \type{#2}%
   \egroup}

#1 is a group name, such as Statement, Operator, or Comment.  #2 may
contain multiple lines, and I don’t know how well this will work on the
TeX side.  It may also contain special characters like {, #, &, and so
on.  Suggestions?

        nikolai

-- 
Nikolai Weibull: now available free of charge at http://bitwi.se/!
Born in Chicago, IL USA; currently residing in Gothenburg, Sweden.
main(){printf(&linux["\021%six\012\0"],(linux)["have"]+"fun"-97);}

[-- Attachment #2: 2context.vim --]
[-- Type: text/plain, Size: 4780 bytes --]

" Vim syntax support file
" Maintainer:       Nikolai Weibull <nikolai@bitwi.se>
" Latest Revision:  2005-12-03

function! s:Format(text, group)
  let formatted = strtrans(a:text)

  " TODO: Replace the reserved ConTeXt characters.

  return "\\highlight[" . a:group . ']{' . formatted . '}'
endfun

" Set up options.
let s:old_title = &title
let s:old_icon = &icon
let s:old_et = &l:et
let s:old_report = &report
let s:old_search = @/
set notitle noicon
setlocal et
set report=1000000

" Split window to create a buffer with the HTML file.
let s:org_bufnr = winbufnr(0)
if expand("%") == ""
  new untitled.tex
else
  new %.tex
endif
let s:new_win = winnr()
let s:org_win = bufwinnr(s:org_bufnr)

" Set up options in the new buffer.
set modifiable
%d
let s:old_paste = &paste
set paste
let s:old_magic = &magic
set magic

" Set up the buffer’s “header”.
exe "normal! a\\startlines\n\e"

" Switch to the original window.
exe s:org_win . 'wincmd w'

" Variables to keep track of the range to convert.
let s:lnum = 1
let s:end = line('$')

" Set up stuff for handling folding.
if has('folding') && !exists('context_ignore_folding')
  let s:foldfillchar = &fillchars[matchend(&fillchars, 'fold:')]
  if s:foldfillchar == ''
    let s:foldfillchar = '-'
  endif
endif

" Set up stuff for handling diffs.
let s:difffillchar = &fillchars[matchend(&fillchars, 'diff:')]
if s:difffillchar == ''
  let s:difffillchar = '-'
endif

" Now, loop over all lines in the range.
while s:lnum <= s:end
  " If there are filler lines for diff mode, show these above the line.
  let s:filler = diff_filler(s:lnum)
  if s:filler > 0
    let s:n = s:filler
    while s:n > 0
      let s:new = repeat(s:difffillchar, 3)

      if s:n > 2 && s:n < s:filler && !exists('context_whole_filler')
	let s:new = s:new . " " . s:filler . ' inserted lines '
	let s:n = 2
      endif

      let s:new = s:new . repeat(s:difffillchar, &columns - strlen(s:new))

      let s:new = s:Format(s:new, 'DiffDelete')
      exe s:new_win . 'wincmd w'
      exe 'normal! a' . s:new . "\n\e"
      exe s:org_win . 'wincmd w'

      let s:n -= 1
    endwhile
    unlet s:n
  endif
  unlet s:filler

  let s:new = ""
  if has('folding') && !exists('context_ignore_folding') && foldclosed(s:lnum) > -1
    let s:new = s:Format(s:new . foldtextresult(s:lnum), 'Folded')
    let s:lnum = foldclosedend(s:lnum)
  else
    let s:line = getline(s:lnum)
    let s:len = strlen(s:line)
    let s:diffattr = diff_hlID(s:lnum, 1)

    let s:col = 1
    while s:col <= s:len || (s:col == 1 && s:diffattr)
      let s:startcol = s:col " The start column for processing text.
      if s:diffattr
	let s:id = diff_hlID(s:lnum, s:col)
	let s:col += 1
	while s:col <= s:len && s:id == diff_hlID(s:lnum, s:col) | let s:col += 1 | endwhile
        if s:len < &columns
	  " Add spaces at the end to mark the changed line.
          let s:line = s:line . repeat(' ', &columns - s:len)
          let s:len = &columns
        endif
      else
	let s:id = synID(s:lnum, s:col, 1)
	let s:col += 1
	while s:col <= s:len && s:id == synID(s:lnum, s:col, 1) | let s:col += 1 | endwhile
      endif

      " Expand tabs.
      let s:expanded = strpart(s:line, s:startcol - 1, s:col - s:startcol)
      let idx = stridx(s:expanded, "\t")
      while idx >= 0
        let i = &ts - (idx + s:startcol - 1) % &ts
        let s:expanded = substitute(s:expanded, '\t', repeat(' ', i), '')
        let idx = stridx(s:expanded, "\t")
      endwhile

      " Output the text with the same synID, with class set to {s:id_name}.
      let s:id = synIDtrans(s:id)
      let s:id_name = synIDattr(s:id, 'name')
      if s:expanded !~ '^\s*$'
        let s:new = s:new . s:Format(s:expanded, s:id_name)
      else
        let s:new = s:new . s:expanded
      endif
    endwhile
  endif

  exe s:new_win . 'wincmd w'
  exe 'normal! a' . s:new . "\n\e"
  exe s:org_win . 'wincmd w'
  let s:lnum = s:lnum + 1
endwhile

" Cleanup.
exe s:new_win . 'wincmd w'
exe "normal! a\\stoplines\n\e"
%s:\s\+$::e
$g/^$/d

" Restore old settings
let &report = s:old_report
let &title = s:old_title
let &icon = s:old_icon
let &paste = s:old_paste
let &magic = s:old_magic
let @/ = s:old_search
exe s:org_win . 'wincmd w'
let &l:et = s:old_et
exe s:new_win . 'wincmd w'

" Save a little bit of memory (worth doing?)
unlet s:old_et s:old_paste s:old_icon s:old_report s:old_title s:old_search
unlet s:lnum s:end s:old_magic
unlet! s:col s:id s:attr s:len s:line s:new s:expandedtab
unlet s:org_win s:new_win s:org_bufnr
if !v:profiling
  delfunc s:Format
endif
silent! unlet s:diffattr s:difffillchar s:foldfillchar

[-- Attachment #3: Type: text/plain, Size: 139 bytes --]

_______________________________________________
ntg-context mailing list
ntg-context@ntg.nl
http://www.ntg.nl/mailman/listinfo/ntg-context

Re: An idea

[Christopher Creutzig]

Nikolai Weibull wrote:

> this than I have is welcome to finish it.  The \highlight command should
> be defined something like this (pseudo-tex-code):
> 
> \pdef\highlight[#1]{#2}%
>   {\bgroup
>    \setupcolorforgroup[#1]%
>    \type{#2}%
>    \egroup}
> 
> #1 is a group name, such as Statement, Operator, or Comment.  #2 may
> contain multiple lines, and I don’t know how well this will work on the
> TeX side.  It may also contain special characters like {, #, &, and so
> on.  Suggestions?

 The most simple one I expect to work would be

\def\highlight[#1]{%
  \bgroup
  \setuphighlightcolors[{#1}]% never forget those {},
%                            % since [] aren't balanced!
  \processinlineverbatim\egroup}% \egroup is an argument to
%                               % \processinlineverbatim here


 (I did have some mildly elaborate low-level code here, but upon
searching for the ConTeXt way of switching of everything, I came upon
\processinlineverbatim, which probably does everything you want.  If it
does not, please come back and I'll send you something more low-level
that should.)


regards,
	Christopher

Re: An idea

[Nikolai Weibull]

[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]

Christopher Creutzig wrote:

> Nikolai Weibull wrote:
> 
> > this than I have is welcome to finish it.  The \highlight command
> > should be defined something like this (pseudo-tex-code):
> > 
> > \pdef\highlight[#1]{#2}%
> >   {\bgroup
> >    \setupcolorforgroup[#1]%
> >    \type{#2}%
> >    \egroup}
> > 
> > #1 is a group name, such as Statement, Operator, or Comment.  #2 may
> > contain multiple lines, and I don’t know how well this will work on
> > the TeX side.  It may also contain special characters like {, #, &,
> > and so on.  Suggestions?
> 
>  The most simple one I expect to work would be
> 
> \def\highlight[#1]{%
>   \bgroup
>   \setuphighlightcolors[{#1}]% never forget those {},
> %                            % since [] aren't balanced!
>   \processinlineverbatim\egroup}% \egroup is an argument to
> %                               % \processinlineverbatim here
> 

OK, check the attachments for what I’ve got so far.  It works well it
seems.  Still need to do more escaping, though.

        nikolai

-- 
Nikolai Weibull: now available free of charge at http://bitwi.se/!
Born in Chicago, IL USA; currently residing in Gothenburg, Sweden.
main(){printf(&linux["\021%six\012\0"],(linux)["have"]+"fun"-97);}

[-- Attachment #2: 2context.vim --]
[-- Type: text/plain, Size: 4646 bytes --]

" Vim syntax support file
" Maintainer:       Nikolai Weibull <nikolai@bitwi.se>
" Latest Revision:  2005-12-04

function! s:Format(text, group)
  let formatted = strtrans(a:text)

  let formatted = substitute(formatted, '{', '\\leftargument', 'g')
  let formatted = substitute(formatted, '}', '\\rightargument', 'g')
  " TODO: Replace the reserved ConTeXt characters.
  " TODO: make sure to escape everything.

  if a:group != ""
    let formatted = "\\highlight[" . a:group . ']{' . formatted . '}'
  endif

  return formatted
endfunction

function! s:Append(text)
  exe s:new_win . 'wincmd w'
  call append(line('$'), a:text)
  exe s:org_win . 'wincmd w'
endfunction

function! s:AppendFillers(filler, diff_fillchar)
  let n = filler
  while n > 0
    let new = repeat(diff_fillchar, 3)

    if n > 2 && n < filler && !exists('context_whole_filler')
      let new .= ' ' . filler . ' inserted lines '
      let n = 2
    endif

    let new .= repeat(diff_fillchar, &columns - strlen(new))
    call s:Append(s:Format(new, 'DiffDelete'))
    let n -= 1
  endwhile
endfunction

function! s:GetFillChar(char, default)
  let fillchar = &fillchars[matchend(&fillchars, a:char . ':')]
  return (fillchar != '' ? fillchar : default)
endfunction

function! s:MainLoop()
  " Set up stuff for handling folding.
  if has('folding') && !exists('context_ignore_folding')
    let fold_fillchar = s:GetFillChar('fold', '-')
  endif

  " Set up stuff for handling diffs.
  let diff_fillchar = s:GetFillChar('diff', '-')

  let lnum = 1
  let last = line('$')

  while lnum <= last
    " If there are filler lines for diff mode, show these above the line.
    let filler = diff_filler(lnum)
    if filler > 0
      call s:AppendFillers(filler, diff_fillchar)
    endif

    let new = ""
    if has('folding') && !exists('context_ignore_folding') && foldclosed(lnum) > -1
      let new = s:Format(new . foldtextresult(lnum), 'Folded')
      let lnum = foldclosedend(lnum)
    else
      let line = getline(lnum)
      let len = strlen(line)
      let diff_attr = diff_hlID(lnum, 1)

      let col = 1
      while col <= len || (col == 1 && diff_attr)
        let start_col = col " The start column for processing text.
        if diff_attr
          let id = diff_hlID(lnum, col)
          let col += 1
          while col <= len && id == diff_hlID(lnum, col) | let col += 1 | endwhile
          if len < &columns
            " Add spaces at the end to mark the changed line.
            let line = line . repeat(' ', &columns - len)
            let len = &columns
          endif
        else
          " TODO: why not use synIDtrans here instead?  That’t give longest
          " possible matches.
          let id = synID(lnum, col, 1)
          let col += 1
          while col <= len && id == synID(lnum, col, 1) | let col += 1 | endwhile
        endif

        " Expand tabs.
        " TODO: make a function? let new .= s:ExpandAndFormat(strpart(...), id)
        let expanded = strpart(line, start_col - 1, col - start_col)
        let idx = stridx(expanded, "\t")
        while idx >= 0
          let i = &ts - (idx + start_col - 1) % &ts
          let expanded = substitute(expanded, '\t', repeat(' ', i), '')
          let idx = stridx(expanded, "\t")
        endwhile

        if expanded =~ '^\s*$'
          let new .= expanded
        else
          let new .= s:Format(expanded, synIDattr(synIDtrans(id), 'name'))
        endif
      endwhile
    endif

    call s:Append(new)
    let lnum = lnum + 1
  endwhile
endfunction

" Set up options.
let s:old_title = &title
let s:old_icon = &icon
let s:old_search = @/
set notitle noicon

" Split window to create a buffer with the ConTeXt file.
let s:org_bufnr = winbufnr(0)
if expand("%") == ""
  new untitled.tex
else
  new %.tex
endif
let s:new_win = winnr()
let s:org_win = bufwinnr(s:org_bufnr)

" Set up the new buffer.
set modifiable
%delete
call setline(1, '\startlines')

" Switch to the original window.
exe s:org_win . 'wincmd w'

call s:MainLoop()

" Cleanup.
exe s:new_win . 'wincmd w'
call append(line('$'), '\stoplines')
silent %s:\s\+$::e

" Restore old settings.
let &title = s:old_title
let &icon = s:old_icon
let @/ = s:old_search

" Save a little bit of memory.  (Is this worth doing?)
unlet s:old_title s:old_icon s:old_search
unlet s:org_bufnr s:new_win s:org_win
if !v:profiling
  delfunction s:Format
  delfunction s:Append
  delfunction s:AppendFillers
  delfunction s:GetFillChar
  delfunction s:MainLoop
endif

[-- Attachment #3: b.c --]
[-- Type: text/x-csrc, Size: 137 bytes --]

/*
 * contents: unknown
 *
 * Copyright © 2005 Nikolai Weibull <nikolai@bitwi.se>
 */


int
main(void)
{
        return 0;
}

[-- Attachment #4: b.c.tex --]
[-- Type: text/x-tex, Size: 1181 bytes --]

\enableregime[utf]

\setupcolors
  [state=start]

\definecolor  [Comment]     [r=0.14509803921568629, g=0.45098039215686275, b=0.14509803921568629]
\definecolor  [Type]        [r=0.37647058823529411, g=0.18431372549019609, b=0.50196078431372548]
\definecolor  [Identifier]  [r=0.18431372549019609, g=0.35294117647058826, b=0.60784313725490191]
\definecolor  [Statement]   [r=0.46274509803921571, g=0.37647058823529411, b=0.12549019607843137]
\definecolor  [Constant]    [r=0.58431372549019611, g=0.086274509803921567, b=0.086274509803921567]

\def\highlight[#1]%
  {\bgroup\startcolor[#1]\processinlineverbatim{\stopcolor\egroup}}

\setuplines[space=yes]

\starttext
\startlines
\setverbatimspaceskip
\frenchspacing
\parindent\zeropoint
\verbatimfont
\highlight[Comment]{/*}
\highlight[Comment]{ * contents: A minimal C program.}
\highlight[Comment]{ *}
\highlight[Comment]{ * Copyright © 2005 Nikolai Weibull <nikolai@bitwi.se>}
 \highlight[Comment]{*/}


\highlight[Type]{int}
\highlight[Identifier]{main}(\highlight[Type]{void})
\leftargument
        \highlight[Statement]{return} \highlight[Constant]{0};
\rightargument
\stoplines
\stoptext

[-- Attachment #5: Type: text/plain, Size: 139 bytes --]

_______________________________________________
ntg-context mailing list
ntg-context@ntg.nl
http://www.ntg.nl/mailman/listinfo/ntg-context

Re: An idea

[Hans Hagen]

Nikolai Weibull wrote:

>Nikolai Weibull wrote:
>
>  
>
>>It would require very little programming.  syntax/2html.vim, which
>>converts the buffer to a HTML document with syntax highlighting, is 526
>>lines in the current CVS incarnation.  A syntax/2context.vim would be
>>even shorter, perhaps 150 to 200 lines.  If I find the time I’ll write
>>something this weekend.  I’m catching a could though, so I might not
>>:-(.
>>    
>>
>
>A splitting headache notwithstanding, here’s a syntax/2context.vim that
>weighs in at 170 lines.  There are still things to do, like figuring out
>how to complement this on the ConTeXt side (i.e., defining \highlight)
>and things will depend on how this is done.  Some sort of \type
>environment would be nice, as it is better to not do escaping of special
>characters on the Vim side.  Someone with better knowledge of how to do
>this than I have is welcome to finish it.  The \highlight command should
>be defined something like this (pseudo-tex-code):
>
>\pdef\highlight[#1]{#2}%
>  {\bgroup
>   \setupcolorforgroup[#1]%
>   \type{#2}%
>   \egroup}
>
>#1 is a group name, such as Statement, Operator, or Comment.  #2 may
>contain multiple lines, and I don’t know how well this will work on the
>TeX side.  It may also contain special characters like {, #, &, and so
>on.  Suggestions?
>  
>

don't use \type, just escape the characters \{ \} etc or use symbolic 
names and use \tttf (or \tt for tje whole code fragment;  \type is 
overkill here an dmy do more hard than good

so you get something

\startVIMhightlighting
\vhl[#1]{....}
\stopVIMhighlighting

with

\def\startVIMhightlighting
  {\startlines \tt}

\def\stopVIMhighlighting
  {\stoplines}

\def\vhl[#1]#2%
  {\color[#1]{#2}}  % maybe even local colors when speed is needed

Even better, be more verbose:

\startVIMhightlighting
\vline{\vword[#1]{....}}
\stopVIMhighlighting

or even:

\startVIMparagraph
\startVIMline\VIMword[#1]{....}  ... \stopVIMline
\stopVIMparagraph

in that case you have way more control (line numbering/treatment, 
spacing etc)

[it's what i did with the scite exporter -- i output highligting in a 
compact XML where even spaces are tagged]

Hans

Hans

Hans

Re: An idea

[Hans Hagen]

Christopher Creutzig wrote:

>Nikolai Weibull wrote:
>
>  
>
>>this than I have is welcome to finish it.  The \highlight command should
>>be defined something like this (pseudo-tex-code):
>>
>>\pdef\highlight[#1]{#2}%
>>  {\bgroup
>>   \setupcolorforgroup[#1]%
>>   \type{#2}%
>>   \egroup}
>>
>>#1 is a group name, such as Statement, Operator, or Comment.  #2 may
>>contain multiple lines, and I don�t know how well this will work on the
>>TeX side.  It may also contain special characters like {, #, &, and so
>>on.  Suggestions?
>>    
>>
>
> The most simple one I expect to work would be
>
>\def\highlight[#1]{%
>  \bgroup
>  \setuphighlightcolors[{#1}]% never forget those {},
>%                            % since [] aren't balanced!
>  \processinlineverbatim\egroup}% \egroup is an argument to
>%                               % \processinlineverbatim here
>
>
> (I did have some mildly elaborate low-level code here, but upon
>searching for the ConTeXt way of switching of everything, I came upon
>\processinlineverbatim, which probably does everything you want.  If it
>does not, please come back and I'll send you something more low-level
>that should.)
>  
>
there is no need for verbatim mode when you convert anyway!

Hans

Re: An idea

[Nikolai Weibull]

Hans Hagen wrote:

[good tips on how to implement the TeX side of things]

Great, thanks!

> [it's what i did with the scite exporter -- i output highligting in a 
> compact XML where even spaces are tagged]

Is there (going to be) any support for this export format in ConTeXt so
that I can just generate this kind of XML instead so that we only do
things once?  I found this in mscite-p.pdf: “The exporter will be
descibed as soon as there are styles for processing the XML code. For
the moment we stick to showing the schema.”  Am I to take it that there
are plans for handling this, but that nothing has been written yet?
I’ll gladly help out if I can.  I was hoping to make an article out of
this, but now it seems that a lot of groundwork has already been done.
(I really need to write some articles if I’m going to be accepted for a
postgraduate position.)

        nikolai

-- 
Nikolai Weibull: now available free of charge at http://bitwi.se/!
Born in Chicago, IL USA; currently residing in Gothenburg, Sweden.
main(){printf(&linux["\021%six\012\0"],(linux)["have"]+"fun"-97);}

Re: An idea

[Hans Hagen]

[-- Attachment #1: Type: text/plain, Size: 1308 bytes --]

Nikolai Weibull wrote:

>Hans Hagen wrote:
>
>[good tips on how to implement the TeX side of things]
>
>Great, thanks!
>
>  
>
>>[it's what i did with the scite exporter -- i output highligting in a 
>>compact XML where even spaces are tagged]
>>    
>>
>
>Is there (going to be) any support for this export format in ConTeXt so
>that I can just generate this kind of XML instead so that we only do
>things once?  I found this in mscite-p.pdf: “The exporter will be
>descibed as soon as there are styles for processing the XML code. For
>the moment we stick to showing the schema.”  Am I to take it that there
>are plans for handling this, but that nothing has been written yet?
>I’ll gladly help out if I can.  I was hoping to make an article out of
>this, but now it seems that a lot of groundwork has already been done.
>(I really need to write some articles if I’m going to be accepted for a
>postgraduate position.)
>  
>
see attached zip   

this is what is produced, writing an xml mapping to context code (i must 
have an example somewhere but cannot find it now)

about an article ... it would be interesting to combine thsi with a kind 
of literate programming

(if cweb would produce better output then it may have become a  bit more 
popular)

Hans

[-- Attachment #2: temp.zip --]
[-- Type: application/zip, Size: 6459 bytes --]

[-- Attachment #3: Type: text/plain, Size: 139 bytes --]

_______________________________________________
ntg-context mailing list
ntg-context@ntg.nl
http://www.ntg.nl/mailman/listinfo/ntg-context

Re: An idea

[Mojca Miklavec]

Nikolai Weibull wrote:
> We were planning on using Vim for preprocessing things like this instead
> of hacking TeX to do the lexing.  I think Mojca (?) had some ideas for
> this and perhaps even a simple implementation, and given that there is
> now structured support for preprocessing in texexec I think that this
> can be done quite easily.  A lot easier than writing something in TeX.

Well, a while ago I experimented a bit with 2context.vim:
http://pub.mojca.org/tex/vim/fromlgrind/,
but this is surely less that Nikolai did with his last attachment (I
didn't check it yet): I only changed two or three lines in 2html.vim,
which was enough to output ConTeXt code out of any source file
supported by vim. The produced PDFs on the page mentioned above are a
bit faked, but just the ConTeXt part of it, since I didn't know how to
make the \highlight[Statement]{content} highlighted in the proper way
and I didn't know how to send the data to vim and then get them back.

Mojca

Re: An idea

[Nikolai Weibull]

[-- Attachment #1: Type: text/plain, Size: 6475 bytes --]

Hans Hagen wrote:

> Nikolai Weibull wrote:

> > Is there (going to be) any support for this export format in ConTeXt
> > so that I can just generate this kind of XML instead so that we only
> > do things once?  I found this in mscite-p.pdf: “The exporter will be
> > descibed as soon as there are styles for processing the XML code.
> > For the moment we stick to showing the schema.”  Am I to take it
> > that there are plans for handling this, but that nothing has been
> > written yet?  I’ll gladly help out if I can.  I was hoping to make
> > an article out of this, but now it seems that a lot of groundwork
> > has already been done.  (I really need to write some articles if I’m
> > going to be accepted for a postgraduate position.)

> see attached zip   

> this is what is produced, writing an xml mapping to context code (i must 
> have an example somewhere but cannot find it now)

(I come off quite negative in this response, but I am not trying to
place blame here, just trying to come up with something better.)

That’s just horrendous.  The ‘n’ attribute on ‘line’ is useless (and why
are empty lines unnumbered?).  You can just as easily use XSLT’s
<number/> to get that effect:

  <xsl:template match="scite:line">
     <xsl:number/>. <xsl:apply-templates/>
  </xsl:template>

Furthermore, the ‘n’ attribute on ‘t’ is equally useless.  How am I to
know what “3” stands for when converting?

Finally, what’s the reason for having ‘s’?  It just complicates matters
immensely:

  <xsl:template match="scite:s">
    <xsl:call-template name="print-spaces">
      <xsl:with-param name="n" select="@n"/>
    </xsl:call-template>
  </xsl:template>

  <xsl:template name="print-spaces">
    <xsl:param name="n"/>
    <xsl:if test="number($n) > 0">
      <xsl:text> </xsl:text>
      <xsl:call-template name="print-spaces">
        <xsl:with-param name="n" select="number($n) - 1"/>
      </xsl:call-template>
    </xsl:if>
  </xsl:template>

The problem is that the ‘n’ attribute has a default of 1, but as there’s
no way to express default values for attributes in Relax-NG (and there’s
no schema at the bogus url http://www.scintila.org/scite.rng) we’d have
to hack it even more.

Could you enlighten me on the merits of having a tag for representing
(sequences of) whitespace?

Here’s a suggestion for a better grammar:

ID.datatype = xsd:ID
LanguageCode.datatype = xsd:language

id.attrib = attribute id { ID.datatype }?
xmlbase.attrib = attribute xml:base { text }?
Core.attrib = id.attrib, xmlbase.attrib
lang.attrib = attribute xml:lang { LanguageCode.datatype }?
I18n.attrib = lang.attrib
Common.attrib = Core.attrib, I18n.attrib

start = file | line | segment | whitespace

file = element file { file.attlist, file.content }
file.attlist = Common.attrib, attribute path { text }, attribute type { text }
file.content = line*

line = element line { line.attlist, line.content }
line.attlist = Common.attrib
line.content = (segment | whitespace)*

segment = element segment { segment.attlist, text }
segment.attlist = Common.attrib, attribute type { text }

whitespace = element whitespace { whitespace.attlist, empty }
whitespace.attlist = Common.attrib

I’m sure that there are things missing and that it can be improved
further, but it’s a good start in my opinion.

What I can’t decide is whether to write things like

<line>
  <segment type="type">int</segment><whitespace/><segment type="normal">a;</segment>
</line>

or

<line>
  <type>int</type><whitespace/><normal>a;</normal>
</line>

The first scheme is easier to extend with more types, if we don’t
validate the value of segment’s ‘type’ attribute (if we do, then both
schemes are equally “easy” to extend), but the second scheme has the
benefit that as much validation as possible can go into the schema and
can thus ease translation.

I envision that the XSLT ConTeXt-converter for this scheme would have
code in one of these two formats:

<xsl:template match="segment">
  \highlight[<xsl:value-of select="@type"/>]{<xsl:value-of select="."/>}
</xsl:template>

or

<xsl:template match="type|normal|comment|...">
  \highlight[<xsl:value-of select="local-name(.)"/>]{<xsl:value-of select="."/>}
</xsl:template>

I really don’t know what I prefer.  The first means that only the source
and destination need to worry about dealing with new types (the
stylesheet can be used unchanged).

> about an article ... it would be interesting to combine thsi with a kind 
> of literate programming

Yes, that’s true.  I used a simple hack to my code written in literate
programming into my master’s thesis.  It worked quite well actually.
I’m sure that this could be combined with the method I used there.

> (if cweb would produce better output then it may have become a  bit
> more popular)

Yes.  I see three problems with CWEB:

1.  The syntax is too complicated
2.  The output is not great
3.  The input isn’t compilable in the source programming language

What my literate programming system did was allow for embedding comments
in the source instead of the other way around (much like how ConTeXt
sources are documented).  This worked very well actually, especially in
the parts of my code that were written in Ruby where one can do things
like

# ¶ OK, so now that we have all the other methods done, it’s time to
# write our final method \Ruby{method}.  It will simply tie together the
# other methods found in \Ruby{Class}:
def Class.method
  ⋮
end

(A sequence of comments where the first comment began with a pilcrow
sign are processed as stuff to send to ConTeXt and the following code
block (basically everything up to the next pilcrow-endowned comment) was
set inside a verbatim environment.)

Anyway, I’ve attached the Relax-NG schema that I deviced.  Tomorrow I’m
going to work on writing an exporter that exports to this format and a
stylesheet that transforms this to ConTeXt code.  If I have time I’ll
also try to set up some environment for this and some generic
preprocessing directives for texexec.

        nikolai

-- 
Nikolai Weibull: now available free of charge at http://bitwi.se/!
Born in Chicago, IL USA; currently residing in Gothenburg, Sweden.
main(){printf(&linux["\021%six\012\0"],(linux)["have"]+"fun"-97);}

[-- Attachment #2: syntax-export.rnc --]
[-- Type: text/plain, Size: 838 bytes --]

ID.datatype = xsd:ID
LanguageCode.datatype = xsd:language

id.attrib = attribute id { ID.datatype }?
xmlbase.attrib = attribute xml:base { text }?
Core.attrib = id.attrib, xmlbase.attrib
lang.attrib = attribute xml:lang { LanguageCode.datatype }?
I18n.attrib = lang.attrib
Common.attrib = Core.attrib, I18n.attrib

start = file | line | segment | whitespace

file = element file { file.attlist, file.content }
file.attlist = Common.attrib, attribute path { text }, attribute type { text }
file.content = line*

line = element line { line.attlist, line.content }
line.attlist = Common.attrib
line.content = (segment | whitespace)*

segment = element segment { segment.attlist, text }
segment.attlist = Common.attrib, attribute type { text }

whitespace = element whitespace { whitespace.attlist, empty }
whitespace.attlist = Common.attrib

[-- Attachment #3: Type: text/plain, Size: 139 bytes --]

_______________________________________________
ntg-context mailing list
ntg-context@ntg.nl
http://www.ntg.nl/mailman/listinfo/ntg-context

Re: An idea

[Hans Hagen]

Nikolai Weibull wrote:

>That’s just horrendous.  The ‘n’ attribute on ‘line’ is useless (and why
>are empty lines unnumbered?).  You can just as easily use XSLT’s
><number/> to get that effect:
>  
>
if i remember right it has to to with the fact that while editing 
inserted lines were treated different (sub numbers and such) so n is not 
om forehand something going up in steps of 1; it's more a status thing

>Furthermore, the ‘n’ attribute on ‘t’ is equally useless.  How am I to
>know what “3” stands for when converting?
>  
>
in scite that's related to styles which have numbers (no symbolic names)

>Finally, what’s the reason for having ‘s’?  It just complicates matters
>immensely:
>
>  <xsl:template match="scite:s">
>    <xsl:call-template name="print-spaces">
>      <xsl:with-param name="n" select="@n"/>
>    </xsl:call-template>
>  </xsl:template>
>  
>
having access to spaces (without the need to xslt the file) can be handy 
if you want to treat them special (e.g. visualize them); keep in mind 
that in xml mode we don't have active character trickery available

actually, it's one of the reasons why i dislike cdata being used for 
verbose content instead of tagged content

<lines>
<line>...</line>
</lines>

is easier to interface to certain typesetting things

btw, when we process xml here, i often have to preprocess the files 
(using ruby and regexps) just in order to add some detail that permits 
proper typesetting

>  <xsl:template name="print-spaces">
>    <xsl:param name="n"/>
>    <xsl:if test="number($n) > 0">
>      <xsl:text> </xsl:text>
>      <xsl:call-template name="print-spaces">
>        <xsl:with-param name="n" select="number($n) - 1"/>
>      </xsl:call-template>
>    </xsl:if>
>  </xsl:template>
>
>The problem is that the ‘n’ attribute has a default of 1, but as there’s
>no way to express default values for attributes in Relax-NG (and there’s
>no schema at the bogus url http://www.scintila.org/scite.rng) we’d have
>to hack it even more.
>
>Could you enlighten me on the merits of having a tag for representing
>(sequences of) whitespace?
>  
>
as said, in order to be able to treat them kind of special

btw, it's all configurable:

export.xml.collapse.spaces=1
export.xml.collapse.lines=1

so .. it's just what i needed (at that time) which was an alternative 
for the latex output mode that i could not use

>Here’s a suggestion for a better grammar:
>
>ID.datatype = xsd:ID
>LanguageCode.datatype = xsd:language
>
>id.attrib = attribute id { ID.datatype }?
>xmlbase.attrib = attribute xml:base { text }?
>Core.attrib = id.attrib, xmlbase.attrib
>lang.attrib = attribute xml:lang { LanguageCode.datatype }?
>I18n.attrib = lang.attrib
>Common.attrib = Core.attrib, I18n.attrib
>
>start = file | line | segment | whitespace
>
>file = element file { file.attlist, file.content }
>file.attlist = Common.attrib, attribute path { text }, attribute type { text }
>file.content = line*
>
>line = element line { line.attlist, line.content }
>line.attlist = Common.attrib
>line.content = (segment | whitespace)*
>
>segment = element segment { segment.attlist, text }
>segment.attlist = Common.attrib, attribute type { text }
>
>whitespace = element whitespace { whitespace.attlist, empty }
>whitespace.attlist = Common.attrib
>
>I’m sure that there are things missing and that it can be improved
>further, but it’s a good start in my opinion.
>
>What I can’t decide is whether to write things like
>
><line>
>  <segment type="type">int</segment><whitespace/><segment type="normal">a;</segment>
></line>
>
>or
>
><line>
>  <type>int</type><whitespace/><normal>a;</normal>
></line>
>
>The first scheme is easier to extend with more types, if we don’t
>validate the value of segment’s ‘type’ attribute (if we do, then both
>schemes are equally “easy” to extend), but the second scheme has the
>benefit that as much validation as possible can go into the schema and
>can thus ease translation.
>
>I envision that the XSLT ConTeXt-converter for this scheme would have
>code in one of these two formats:
>
><xsl:template match="segment">
>  \highlight[<xsl:value-of select="@type"/>]{<xsl:value-of select="."/>}
></xsl:template>
>
>or
>
><xsl:template match="type|normal|comment|...">
>  \highlight[<xsl:value-of select="local-name(.)"/>]{<xsl:value-of select="."/>}
></xsl:template>
>
>I really don’t know what I prefer.  The first means that only the source
>and destination need to worry about dealing with new types (the
>stylesheet can be used unchanged).
>  
>
it depends a bit on what an editor can provide; the scite xml output is 
rather connected to its lexer, and since each lexer is different there 
is no strict relationship between a number and e.g. a type. As a result, 
stylesheets need to have that kind of (conversion) knowledge.

it's not clear to me how you want to use these code snippets

- if you're going to copy them into your tex document, i'd leave them in 
xml format, i.e. have a mixed tex/xml doc (no need for conversion since 
we're dealing with rather simple xml->tex mapping sthat can be done by 
context itself)
- if you keep them in the source file, you need a way to filter them 
into your main text document (this is what i do when i nedd to document 
for stylesheets and such: i filter the pieces i needed while typesetting)

>>about an article ... it would be interesting to combine thsi with a kind 
>>of literate programming
>>    
>>
>
>Yes, that’s true.  I used a simple hack to my code written in literate
>programming into my master’s thesis.  It worked quite well actually.
>I’m sure that this could be combined with the method I used there.
>  
>
so how does this work?

some text

[reference to code snippet]

some more text

where you merge in the code snippets before processing?

>  
>
>>(if cweb would produce better output then it may have become a  bit
>>more popular)
>>    
>>
>
>Yes.  I see three problems with CWEB:
>
>1.  The syntax is too complicated
>2.  The output is not great
>3.  The input isn’t compilable in the source programming language
>
>What my literate programming system did was allow for embedding comments
>in the source instead of the other way around (much like how ConTeXt
>sources are documented).  This worked very well actually, especially in
>the parts of my code that were written in Ruby where one can do things
>like
>
># ¶ OK, so now that we have all the other methods done, it’s time to
># write our final method \Ruby{method}.  It will simply tie together the
># other methods found in \Ruby{Class}:
>  
>
why not

# write our final method \RubyM[Class.method].  It will simply tie together the
# other methods found in \RubyC[Class.method]:

and let that info be auto-inserted? Now you have to edit three (probably 
more) places when you change the name of the method.

>def Class.method
>  ⋮
>end
>
>(A sequence of comments where the first comment began with a pilcrow
>sign are processed as stuff to send to ConTeXt and the following code
>block (basically everything up to the next pilcrow-endowned comment) was
>set inside a verbatim environment.)
>
>Anyway, I’ve attached the Relax-NG schema that I deviced.  Tomorrow I’m
>going to work on writing an exporter that exports to this format and a
>stylesheet that transforms this to ConTeXt code.  If I have time I’ll
>also try to set up some environment for this and some generic
>preprocessing directives for texexec.
>  
>
ok

Hans

Re: An idea

[Christopher Creutzig]

Hans Hagen wrote:

> there is no need for verbatim mode when you convert anyway!

 Technically true, but if a simple \processinlineverbatim solves the
same problem as yet-to-write code in almost any language, I's go for the
former, since it does already exist.  Other than that, I'd use ConTeXt's
xml parser, since it requires much less escaping; & and < should suffice.


Regards,
	Christopher