Mark IV Syntax highlighting support

Mark IV Syntax highlighting support

[Renaud Aubin]

Hi folks,

Where to start with this topic (current state in Mark IV, roadmap, ...) ?

Renaud
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Mark IV Syntax highlighting support

[Renaud Aubin]

./tex/texmf-context/tex/context/base/verb-.*\.lua !

Renaud Aubin a écrit :
> Hi folks,
>
> Where to start with this topic (current state in Mark IV, roadmap, ...) ?
>
> Renaud
> ___________________________________________________________________________________ 
>
> If your question is of interest to others as well, please add an entry 
> to the Wiki!
>
> maillist : ntg-context@ntg.nl / 
> http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
> archive  : https://foundry.supelec.fr/projects/contextrev/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________ 
>
>
>

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Mark IV Syntax highlighting support

[Matthijs Kooijman]

[-- Attachment #1.1: Type: text/plain, Size: 740 bytes --]

Hey all,

> ./tex/texmf-context/tex/context/base/verb-.*\.lua !
Actually, in newer versions (since a month or so), that's pret-*.lua.

>> Where to start with this topic (current state in Mark IV, roadmap, ...) ?
I've been struggling with this a bit as well lately and found there was next
to no documentation. I've managed to create my own pretty printer, and tried
to update the docs a bit. See [1] and [2] on the wiki for more info. There are
still some details I don't get (why is there a line() hook, what is all this
nesting stuff, etc.), so feel free to find out more and improve these pages!

Gr.

Matthijs

[1]: http://wiki.contextgarden.net/Verbatim_text#Your_own_formatter
[2]: http://wiki.contextgarden.net/Custom_pretty_printer

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 487 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Mark IV Syntax highlighting support

[Renaud Aubin]

[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]

Hi folks !

I have just finished a first draft of pret-xml.lua:
- I need to improve its coverage (html);
- DTD declaration is not yet managed;
- I do nothing about re-indentation since I prefer to format my xml 
sample by myself (by hand or using xmllint);
- That's my first try to lua/luatex (after several months of Java...) so 
my definitions (esp. for functions) is not clean at all (you have been 
warned);

I have done it the "trial/error" way without much design reflexion just 
for fun (I'm on vacation)... I'm not sure it will be reusable...

Take all your test files and please report issues (or give me advices)! ;)

Thank you Matthijs!

Regards,

Renaud

Matthijs Kooijman a écrit :
> Hey all,
>
>> ./tex/texmf-context/tex/context/base/verb-.*\.lua !
> Actually, in newer versions (since a month or so), that's pret-*.lua.
>
>>> Where to start with this topic (current state in Mark IV, roadmap, ...) ?
> I've been struggling with this a bit as well lately and found there was next
> to no documentation. I've managed to create my own pretty printer, and tried
> to update the docs a bit. See [1] and [2] on the wiki for more info. There are
> still some details I don't get (why is there a line() hook, what is all this
> nesting stuff, etc.), so feel free to find out more and improve these pages!
>
> Gr.
>
> Matthijs
>
> [1]: http://wiki.contextgarden.net/Verbatim_text#Your_own_formatter
> [2]: http://wiki.contextgarden.net/Custom_pretty_printer
>
> ------------------------------------------------------------------------
>
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
>
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
> archive  : https://foundry.supelec.fr/projects/contextrev/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________


[-- Attachment #2: pret-xml.lua --]
[-- Type: text/x-lua, Size: 6745 bytes --]

if not modules then modules = { } end modules ['pret-xml'] = {
    version   = 1.001,
    comment   = "from pret-.*\.lua, http://wiki.contextgarden.net/Verbatim_text#Your_own_formatter and http://wiki.contextgarden.net/Custom_pretty_printer",
    author    = "Renaud AUBIN",
    copyright = "to be defined",
    license   = "to be defined"
}

local utf = unicode.utf8

local utfcharacters, utfvalues = string.utfcharacters, string.utfvalues
local utfbyte, utffind = utf.byte, utf.find
local byte, sub, find, gfind, match = string.byte, string.sub, string.find, string.gfind, string.match
local texsprint, texwrite = tex.sprint, tex.write
local ctxcatcodes = tex.ctxcatcodes

local change_state, finish_state = buffers.change_state, buffers.finish_state

buffers.visualizers.xml             = buffers.visualizers.xml             or { }
buffers.visualizers.xml.identifiers = buffers.visualizers.xml.identifiers or { }

incomment = false

local colors = {
   "prettyone",
   "prettytwo",
   "prettythree",
   "prettyfour",
   "prettyfive",
   "prettysix",
   "prettyseven"
}

local states = {
   ['namespace']        =  1,
   ['element']          =  2,
   ['attrname']         =  3,
   ['attrvalue']        =  4,
   ['prolog']           =  5,
   ['comment']          =  6,
   ['pi']               =  7
}

local function flush_text(str)
   for c in utfcharacters(str) do
      if c == " " then texsprint(ctxcatcodes,"\\obs") else texwrite(c) end
   end
end

local function process_attribute(str)
   local name, value, state = "", "", 0
   buffers.currentcolors = colors
   local name,value = match(str,"^(.-)=(\".-\")$")

   state = change_state(states['attrname'], state)
   flush_text(name)
   state = finish_state(state)
   texwrite("=")
   state = change_state(states['attrvalue'], state)
   flush_text(value)
   state = finish_state(state)
end

local function process_namespace(str)
   local state = 0
   buffers.currentcolors = colors
   
   state = change_state(states['namespace'], state)
   flush_text(str)
   state = finish_state(state)
   texwrite(":")
end

local function process_element(str)
   local state = 0
   buffers.currentcolors = colors
   
   state = change_state(states['element'], state)
   flush_text(str)
   state = finish_state(state)
end

local function process_comment(str)
   local state = 0
   buffers.currentcolors = colors
   
   state = change_state(states['comment'], state)
   flush_text(str)
   state = finish_state(state)
end

function process_tag(str)
   local name, value, state = "", "", 0
   buffers.currentcolors = colors

   if match(str,"^<%?xml.-%?>$") then
      local prolog = match(str,"^<%?xml(.-)%?>$")      
      state = change_state(states['pi'], state)
      texwrite("<?")
      state = finish_state(state)
      state = change_state(states['prolog'], state)
      texwrite("xml")
      state = finish_state(state)
      repeat prolog=recursive_processing(prolog) until prolog==nil
      state = change_state(states['pi'], state)
      texwrite("?>")
      state = finish_state(state)
   elseif match(str,"^<!%-%-.-%-%->$") then
      process_comment(str)
   elseif match(str,"^<[%a_]%w-%s*/?>$") or match(str,"^</?[%a_]%w-%s*>$") then
      local btag, name, etag = match(str,"^(</?)([%a_]%w-%s*)(/?>)$")
      texwrite(btag)
      process_element(name)
      texwrite(etag) 
   elseif match(str,"^<%a%w-:[%a_]%w-%s*/?>$") or match(str,"^</?%a%w-:[%a_]%w-%s*>$") then
      local btag, ns, name, etag = match(str,"^(</?)(%a%w-):([%a_]%w-%s*)(/?>)$")
      texwrite(btag)
      process_namespace(ns)
      process_element(name)
      texwrite(etag)
   elseif match(str,"^<%a%w-:[%a_][%w%-]-%s.*/?>$") or match(str,"^</?%a%w-:[%a_][%w%-]-%s.*>$") then
      local btag, ns, name, attributes, etag = match(str,"^(</?)(%a%w-):([%a_][%w%-]-%s)(.*)(/?>)$")
      texwrite(btag)
      process_namespace(ns)
      process_element(name)
      repeat attributes=recursive_processing(attributes) until attributes==nil
      texwrite(etag)
   elseif match(str,"^<[%a_]%w-%s.*/?>$") or match(str,"^</?[%a_]%w-%s.*>$") then
      local btag, name, attributes, etag = match(str,"^(</?)([%a_]%w-%s)(.*)(/?>)$")
      texwrite(btag)
      process_element(name)
      repeat attributes=recursive_processing(attributes) until attributes==nil
      texwrite(etag)
   end
end

function process_uncomplete_starttag(str)
   local name, value, state = "", "", 0
   buffers.currentcolors = colors

   if match(str,"^<%a%w-:[%a_]%w-%s.*$") then
      local btag, ns, name, attributes = match(str,"^(<)(%a%w-):([%a_]%w-%s)(.*)$")
      texwrite(btag)
      process_namespace(ns)
      process_element(name)
      repeat attributes=recursive_processing(attributes) until attributes==nil
   elseif match(str,"^<[%a_]%w-%s.*$") then
      local btag, name, attributes = match(str,"^(<)([%a_]%w-%s)(.*)$")
      texwrite(btag)
      process_element(name)
      repeat attributes=recursive_processing(attributes) until attributes==nil
   end
end

function recursive_processing(str)
   local first, second, third, attribute = "", "", "", ""
   
   if match(str, "^(.-)(%-%->)(.*)$") then
      -- stop multiline comment
      first, second, third = match(str, "^(.-)(%-%->)(.*)$")
      process_comment(first..second)
      incomment = false
      return third
   elseif incomment then
      process_comment(str)
   elseif match(str, "^(.-)(%b<>)(.*)$") then
      -- standard tag
      first, second, third = match(str, "^(.-)(%b<>)(.*)$")
      repeat first=recursive_processing(first) until first==nil
      process_tag(second)
      return third
   elseif match(str, "^(.-)(<[%a_]%w-:?%w-[%s%c])(.*)$") then
      -- multiline starttag
      first, second, third = match(str, "^(.-)(<[%a_]%w-:?%w-[%s%c])(.*)$")
      repeat first=recursive_processing(first) until first==nil
      process_uncomplete_starttag(second)
      return third
   -- elseif match(str, "^(.-)(<%?xml)(.*)$") then
   -- TODO manage multiline prolog
   -- first, second, third = match(str, "^(.-)(<%?xml)(.*)$")
   -- return third
   elseif match(str, "^(.-)(<!%-%-)(.*)$") then
      -- start multiline comment
      first, second, third = match(str, "^(.-)(<!%-%-)(.*)$")
      repeat first=recursive_processing(first) until first==nil
      process_comment(second..third)
      incomment = true
      return nil
   elseif match(str, "^(.-=\".-\")(.*)$") then
      -- attribute
      local attribute, after = match(str, "^(.-=\".-\")(.*)$")
      process_attribute(attribute)
      return after
   elseif match(str, "^(.*)$") then
      -- other
      first = match(str, "^(.*)$")
      flush_text(str)
   end
   return nil
end

function buffers.visualizers.xml.flush_line(str,nested)

   repeat str=recursive_processing(str) until str==nil
end

[-- Attachment #3: Type: text/plain, Size: 487 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Mark IV Syntax highlighting support

[Hans Hagen]

Renaud Aubin wrote:
> Hi folks !
> 
> I have just finished a first draft of pret-xml.lua:
> - I need to improve its coverage (html);
> - DTD declaration is not yet managed;
> - I do nothing about re-indentation since I prefer to format my xml 
> sample by myself (by hand or using xmllint);
> - That's my first try to lua/luatex (after several months of Java...) so 
> my definitions (esp. for functions) is not clean at all (you have been 
> warned);
> 
> I have done it the "trial/error" way without much design reflexion just 
> for fun (I'm on vacation)... I'm not sure it will be reusable...
> 
> Take all your test files and please report issues (or give me advices)! ;)

syntax highlighting is kind of tricky esp since we can have either 
correct syntax or faulty (can be educational)

this is why the pretty printers work in a rather dumb non validating 
mode; however, in mkii we also have parser based variants for xml pretty 
printing

in mkiv i will provide only a few pretty printers, i.e. the ones needed 
for my manuals; these will be organized under pre-*

in addition i will provide a user installable variant but i need to 
think about it a bit; we also need more verbose names then so that there 
can be multiple pretty printers per file type; (i will also make a basic 
configureable mapper but that's for later)

so, we can have both validating and non validating pretty printing and 
multiple variants

in any case .. keep making and testing them .. i'll try to cook up the 
plug in mechanism and naming scheme asap

Hans


ps. dtd declaration etc ... i shall provide a pretty printer based on th 
ebuild in xml parser (kind of like what we have in mkii); of course this 
leaves along non validated coloring



-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________