ConTeXt Minimals parsing of .tui broken with ruby 1.9.1 in Windows

[Jose Augusto <jasaugusto@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 5397 bytes --]

Hello all,

I want to report a problem that is either in ConTeXt, or in ruby 1.9.1
(last version of ruby). More probably, the problem has to do with ruby
handling non-ASCII characters. I have no means of trying Linux, Solaris,
etc...
Anyone using ConTeXt with ruby 1.9.1 will face it probably (at least in
Windows :-)

The problem happens with all files, even with the simple "Hello":

    \starttext
    Hello World
    \stoptext

After installing ConTeXt Minimals (the devel version) yesterday,
I ran the above example with ruby 1.9.1-p129 in Windows
(both Win 2000 and XP show the problem).

Meanwhile I compiled and tried several versions of Ruby, and found the
following pattern of problems:

ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-mingw32]     PROBLEM
ruby 1.9.1p0 (2009-01-30 revision 21907) [i386-mingw32]        PROBLEM
ruby 1.9.0 (2008-10-04 revision 19669) [i386-mingw32]          No problem
ruby 1.8.7 (2009-06-12 patchlevel 174) [i386-mingw32]          No problem
ruby 1.8.6 (2009-06-08 patchlevel 369) [i386-mingw32]          No problem

So, whatever it is, it is broken with ruby 1.9.1.
All the versions of ruby were compiled in Windows using the mingw toolchain,
with GCC 3.4.5.


>>>> Here is the description of what happens   <<<<

When texutil parses the .tui file, I get the following (see comments after
this text output):

---------------------------------------------
    ...........................
    Output written on con-hello1.pdf (1 page, 21759 bytes).
    Transcript written on con-hello1.log.
    TeXUtil | parsing file con-hello1.tui
    TeXUtil | debug 1 jasa #<File:0x1271d18>
    xxx c \thisissectionseparator{-}
    xxx c \thisisutilityversion{2008.10.14}
    xxx c \thisisbytesequence{?+Ç}
    TeXUtil | fatal error in parsing con-hello1.tui
    TeXUtil | shortcuts : 0
    TeXUtil | expansions: 0
    TeXUtil | reductions: 0
    TeXUtil | divisions : 0
    TeXUtil | loaded files: 0
    TeXUtil | temporary files: 0
    TeXUtil | commands: 2
    TeXUtil | programs: 0
    TeXUtil | tuo file saved
    TeXExec | runtime: 2.703125
--------------------------------------------

The lines with "debug 1 jasa" and starting with "xxx" result from
the simple "debug code" I inserted in the file texutil.rb to find the
problematic line. The error happens when the following ruby code
is executed: (the extra  debug lines have a mark # jasa )

---------------------- texutil.rb (snippet, around line 1025)
----------------------

        def loaded(filename)
            begin
                tuifile = File.suffixed(filename,'tui')
                if FileTest.file?(tuifile) then
                    report("parsing file #{tuifile}")
                    if f = open(tuifile) then
            report("debug 1 jasa #{f}")  # jasa
                        f.each do |line|
                print "xxx #{line}"  # jasa
                            case line.chomp
                                when /^f (.*)$/o then
@plugins.reader('MyFiles',    $1.splitdata)
                                when /^c (.*)$/o then
@plugins.reader('MyCommands', [$1])
                                when /^e (.*)$/o then
@plugins.reader('MyExtras',   $1.splitdata)
                                when /^s (.*)$/o then
@plugins.reader('MySynonyms', $1.splitdata)
                                when /^r (.*)$/o then
@plugins.reader('MyRegisters',$1.splitdata)
                                when /^p (.*)$/o then
@plugins.reader('MyPlugins',  $1.splitdata)
                                when /^x (.*)$/o then
@plugins.reader('MyKeys',     $1.splitdata)
                                when /^r (.*)$/o then # nothing, not handled
here
                            else
                                # report("unknown entry #{line[0,1]} in line
#{line.chomp}")
                            end
                end
                        f.close
                    end
                else
                    report("unable to locate #{tuifile}")
                end
            rescue
                report("fatal error in parsing #{tuifile}")
                @filename = 'texutil'
            else
                @filename = filename
            end
        end

---------------------------------------------------------------

>From the debugging lines that are expelled, it is clear that the line in the
.tui file that triggers the problem is:

    c \thisisbytesequence{ ...non-ASCII codes... }

and, precisely, it s the second line of the 'case':

    when /^c (.*)$/o then @plugins.reader('MyCommands', [$1])

which processes the .tui line and triggers the 'rescue' clause. So I think
the problem lies in the "digestion"
of non-ASCII characters by the last version of Ruby.

I don't know what is the meaning of the \thisisbytesequence line in ConTeXt
and the maening of those non-ASCII chars. I followed the
@plugins.reader('MyCommands', [$1])
and figured out that what raises the exception happens before the
@plugins.reader method,
since it is never reached when the  "\thisisbytesequence" line is processed.

To finish, I think this can be of interest for Ruby guys, but is also
breaking ConTeXt for anyone with ruby 1.9.1 installed in Windows.

Thanks,

Jose Soares Augusto

jasaugusto@gmail.com
jasa@fisica.fc.ul.pt

[-- Attachment #1.2: Type: text/html, Size: 6035 bytes --]

[-- Attachment #2: Type: text/plain, Size: 487 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________