From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.comp.tex.context/51777 Path: news.gmane.org!not-for-mail From: Jose Augusto Newsgroups: gmane.comp.tex.context Subject: ConTeXt Minimals parsing of .tui broken with ruby 1.9.1 in Windows Date: Mon, 13 Jul 2009 18:38:56 +0100 Message-ID: <1781d7210907131038o6c006ac4q8372f5d7f3ade447@mail.gmail.com> Reply-To: mailing list for ConTeXt users NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1920186617==" X-Trace: ger.gmane.org 1247555492 6324 80.91.229.12 (14 Jul 2009 07:11:32 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 14 Jul 2009 07:11:32 +0000 (UTC) To: ntg-context@ntg.nl Original-X-From: ntg-context-bounces@ntg.nl Tue Jul 14 09:11:23 2009 Return-path: Envelope-to: gctc-ntg-context-518@m.gmane.org Original-Received: from ronja.vet.uu.nl ([131.211.172.88] helo=ronja.ntg.nl) by lo.gmane.org with esmtp (Exim 4.50) id 1MQcAc-0004OQ-Ez for gctc-ntg-context-518@m.gmane.org; Tue, 14 Jul 2009 09:11:22 +0200 Original-Received: from localhost (localhost [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id A86841FB8E; Tue, 14 Jul 2009 09:11:21 +0200 (CEST) Original-Received: from ronja.ntg.nl ([127.0.0.1]) by localhost (smtp.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 29365-02; Tue, 14 Jul 2009 09:10:32 +0200 (CEST) Original-Received: from ronja.vet.uu.nl (localhost [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id E99AA1FB1D; Tue, 14 Jul 2009 09:10:31 +0200 (CEST) Original-Received: from localhost (localhost [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id C51371FADF for ; Mon, 13 Jul 2009 19:39:09 +0200 (CEST) Original-Received: from ronja.ntg.nl ([127.0.0.1]) by localhost (smtp.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 12860-07 for ; Mon, 13 Jul 2009 19:38:57 +0200 (CEST) Original-Received: from filter1-ams.mf.surf.net (filter1-ams.mf.surf.net [192.87.102.69]) by ronja.ntg.nl (Postfix) with ESMTP id 72F931FA4C for ; Mon, 13 Jul 2009 19:38:57 +0200 (CEST) Original-Received: from mail-bw0-f207.google.com (mail-bw0-f207.google.com [209.85.218.207]) by filter1-ams.mf.surf.net (8.14.3/8.14.3/Debian-5) with ESMTP id n6DHcuDE029296 for ; Mon, 13 Jul 2009 19:38:56 +0200 Original-Received: by bwz3 with SMTP id 3so2279697bwz.2 for ; Mon, 13 Jul 2009 10:38:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=1qq/9eSxob5jh6phJIWyFR7zAsoyQpzOYw6SEgXOIVA=; b=M6AZCgMX+YG9aXOQuWot78IbrmnSMc0ndC0LpmtXzZ9+8dbxtQK+ecwj47VuQ05+C6 zaA5MTSZP2wXbeKiUU+G4VKIiFahcMVqh5DAY6s31QZCXdecG1s4yHRq5Pj8Gd6XSi3R hshi6eg230/YNMmLOtbFZ91hjp7g94p6/uSAY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=Tq1EXxTO4GP7FkqvbHbhKoC6sUYWswb9PJdd/z6hhL4x2ohtA/gVuQp23wWdNiefZ9 ru1g6cLWcENkycCrKjfeGVRunM27rzgQ7zUgZcuriD8vft1dDcIUIwljNeLvJw+ddZAz GM3FxREWdzVy13qLEzLixMfrd47qRxuPNRZ6M= Original-Received: by 10.103.52.13 with SMTP id e13mr2842310muk.52.1247506736150; Mon, 13 Jul 2009 10:38:56 -0700 (PDT) X-Bayes-Prob: 0.0001 (Score 0, tokens from: @@RPTN) X-CanIt-Geo: ip=209.85.218.207; country=US; region=CA; city=Mountain View; postalcode=94043; latitude=37.4192; longitude=-122.0574; metrocode=807; areacode=650; http://maps.google.com/maps?q=37.4192,-122.0574&z=6 X-CanItPRO-Stream: uu:ntg-context@ntg.nl (inherits from uu:default, base:default) X-Canit-Stats-ID: 260149080 - 6af171e421b9 - 20090713 X-Scanned-By: CanIt (www . roaringpenguin . com) on 192.87.102.69 X-Virus-Scanned: amavisd-new at ntg.nl X-Mailman-Approved-At: Tue, 14 Jul 2009 09:10:30 +0200 X-BeenThere: ntg-context@ntg.nl X-Mailman-Version: 2.1.11 Precedence: list List-Id: mailing list for ConTeXt users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: ntg-context-bounces@ntg.nl Errors-To: ntg-context-bounces@ntg.nl X-Virus-Scanned: amavisd-new at ntg.nl Xref: news.gmane.org gmane.comp.tex.context:51777 Archived-At: --===============1920186617== Content-Type: multipart/alternative; boundary=0016364166ffdcfdf2046e99cb20 --0016364166ffdcfdf2046e99cb20 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello all, I want to report a problem that is either in ConTeXt, or in ruby 1.9.1 (last version of ruby). More probably, the problem has to do with ruby handling non-ASCII characters. I have no means of trying Linux, Solaris, etc... Anyone using ConTeXt with ruby 1.9.1 will face it probably (at least in Windows :-) The problem happens with all files, even with the simple "Hello": \starttext Hello World \stoptext After installing ConTeXt Minimals (the devel version) yesterday, I ran the above example with ruby 1.9.1-p129 in Windows (both Win 2000 and XP show the problem). Meanwhile I compiled and tried several versions of Ruby, and found the following pattern of problems: ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-mingw32] PROBLEM ruby 1.9.1p0 (2009-01-30 revision 21907) [i386-mingw32] PROBLEM ruby 1.9.0 (2008-10-04 revision 19669) [i386-mingw32] No problem ruby 1.8.7 (2009-06-12 patchlevel 174) [i386-mingw32] No problem ruby 1.8.6 (2009-06-08 patchlevel 369) [i386-mingw32] No problem So, whatever it is, it is broken with ruby 1.9.1. All the versions of ruby were compiled in Windows using the mingw toolchain= , with GCC 3.4.5. >>>> Here is the description of what happens <<<< When texutil parses the .tui file, I get the following (see comments after this text output): --------------------------------------------- ........................... Output written on con-hello1.pdf (1 page, 21759 bytes). Transcript written on con-hello1.log. TeXUtil | parsing file con-hello1.tui TeXUtil | debug 1 jasa # xxx c \thisissectionseparator{-} xxx c \thisisutilityversion{2008.10.14} xxx c \thisisbytesequence{?+=C7} TeXUtil | fatal error in parsing con-hello1.tui TeXUtil | shortcuts : 0 TeXUtil | expansions: 0 TeXUtil | reductions: 0 TeXUtil | divisions : 0 TeXUtil | loaded files: 0 TeXUtil | temporary files: 0 TeXUtil | commands: 2 TeXUtil | programs: 0 TeXUtil | tuo file saved TeXExec | runtime: 2.703125 -------------------------------------------- The lines with "debug 1 jasa" and starting with "xxx" result from the simple "debug code" I inserted in the file texutil.rb to find the problematic line. The error happens when the following ruby code is executed: (the extra debug lines have a mark # jasa ) ---------------------- texutil.rb (snippet, around line 1025) ---------------------- def loaded(filename) begin tuifile =3D File.suffixed(filename,'tui') if FileTest.file?(tuifile) then report("parsing file #{tuifile}") if f =3D open(tuifile) then report("debug 1 jasa #{f}") # jasa f.each do |line| print "xxx #{line}" # jasa case line.chomp when /^f (.*)$/o then @plugins.reader('MyFiles', $1.splitdata) when /^c (.*)$/o then @plugins.reader('MyCommands', [$1]) when /^e (.*)$/o then @plugins.reader('MyExtras', $1.splitdata) when /^s (.*)$/o then @plugins.reader('MySynonyms', $1.splitdata) when /^r (.*)$/o then @plugins.reader('MyRegisters',$1.splitdata) when /^p (.*)$/o then @plugins.reader('MyPlugins', $1.splitdata) when /^x (.*)$/o then @plugins.reader('MyKeys', $1.splitdata) when /^r (.*)$/o then # nothing, not handle= d here else # report("unknown entry #{line[0,1]} in lin= e #{line.chomp}") end end f.close end else report("unable to locate #{tuifile}") end rescue report("fatal error in parsing #{tuifile}") @filename =3D 'texutil' else @filename =3D filename end end --------------------------------------------------------------- >>From the debugging lines that are expelled, it is clear that the line in th= e .tui file that triggers the problem is: c \thisisbytesequence{ ...non-ASCII codes... } and, precisely, it s the second line of the 'case': when /^c (.*)$/o then @plugins.reader('MyCommands', [$1]) which processes the .tui line and triggers the 'rescue' clause. So I think the problem lies in the "digestion" of non-ASCII characters by the last version of Ruby. I don't know what is the meaning of the \thisisbytesequence line in ConTeXt and the maening of those non-ASCII chars. I followed the @plugins.reader('MyCommands', [$1]) and figured out that what raises the exception happens before the @plugins.reader method, since it is never reached when the "\thisisbytesequence" line is processed= . To finish, I think this can be of interest for Ruby guys, but is also breaking ConTeXt for anyone with ruby 1.9.1 installed in Windows. Thanks, Jose Soares Augusto jasaugusto@gmail.com jasa@fisica.fc.ul.pt --0016364166ffdcfdf2046e99cb20 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello all,

I want to report a problem that is either in ConTeXt, or = in ruby 1.9.1
(last version of ruby). More probably, the problem has to = do with ruby
handling non-ASCII characters. I have no means of trying Li= nux, Solaris, etc...
Anyone using ConTeXt with ruby 1.9.1 will face it probably (at least in Win= dows :-)

The problem happens with all files, even with the simple &q= uot;Hello":

=A0=A0=A0 \starttext
=A0=A0=A0 Hello World
= =A0=A0=A0 \stoptext

After installing ConTeXt Minimals (the devel version) yesterday,
I r= an the above example with ruby 1.9.1-p129 in Windows
(both Win 2000 and= XP show the problem).

Meanwhile I compiled and tried several versi= ons of Ruby, and found the
following pattern of problems:

ruby 1.9.1p129 (2009-05-12 revision 2= 3412) [i386-mingw32]=A0=A0=A0=A0 PROBLEM
ruby 1.9.1p0 (2009-01-30 revisi= on 21907) [i386-mingw32]=A0=A0=A0=A0=A0=A0=A0 PROBLEM
ruby 1.9.0 (2008-1= 0-04 revision 19669) [i386-mingw32]=A0=A0=A0=A0=A0=A0=A0=A0=A0 No problem ruby 1.8.7 (2009-06-12 patchlevel 174) [i386-mingw32]=A0=A0=A0=A0=A0=A0=A0= =A0=A0 No problem
ruby 1.8.6 (2009-06-08 patchlevel 369) [i386-mingw32]= =A0=A0=A0=A0=A0=A0=A0=A0=A0 No problem

So, whatever it is, it is bro= ken with ruby 1.9.1.
All the versions of ruby were compiled in Windows u= sing the mingw toolchain, with GCC 3.4.5.


>>>> Here is the description of what happens=A0=A0 <= <<<

When texutil parses the .tui file, I get the following = (see comments after this text output):

----------------------------= -----------------
=A0=A0=A0 ...........................
=A0=A0=A0 Output written on con-he= llo1.pdf (1 page, 21759 bytes).
=A0=A0=A0 Transcript written on con-hell= o1.log.
=A0=A0=A0 TeXUtil | parsing file con-hello1.tui
=A0=A0=A0 TeX= Util | debug 1 jasa #<File:0x1271d18>
=A0=A0=A0 xxx c \thisissectionseparator{-}
=A0=A0=A0 xxx c \thisisutilit= yversion{2008.10.14}
=A0=A0=A0 xxx c \thisisbytesequence{?+=C7}
=A0= =A0=A0 TeXUtil | fatal error in parsing con-hello1.tui
=A0=A0=A0 TeXUtil= | shortcuts : 0
=A0=A0=A0 TeXUtil | expansions: 0
=A0=A0=A0 TeXUtil | reductions: 0
=A0=A0=A0 TeXUtil | divisions : 0
= =A0=A0=A0 TeXUtil | loaded files: 0
=A0=A0=A0 TeXUtil | temporary files:= 0
=A0=A0=A0 TeXUtil | commands: 2
=A0=A0=A0 TeXUtil | programs: 0=A0=A0=A0 TeXUtil | tuo file saved
=A0=A0=A0 TeXExec | runtime: 2.703125
----------------------------------= ----------

The lines with "debug 1 jasa" and starting with= "xxx" result from
the simple "debug code" I inserte= d in the file texutil.rb to find the
problematic line. The error happens when the following ruby code
is exe= cuted: (the extra=A0 debug lines have a mark # jasa )

--------------= -------- texutil.rb (snippet, around line 1025) ----------------------

=A0=A0=A0=A0=A0=A0=A0 def loaded(filename)
=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0 begin
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 tuifile = =3D File.suffixed(filename,'tui')
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 if FileTest.file?(tuifile) then
=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 report("parsing file #{tuifile}"= ;)
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 if f =3D open(tui= file) then
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 report("debug 1 jasa #{f}&= quot;)=A0 # jasa
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 f.each do |line|
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0= print "xxx #{line}"=A0 # jasa
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 case line.chomp
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 when /^f (.*)$/o then @plugins.reader('MyFiles',= =A0=A0=A0 $1.splitdata)
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 when /^c (.*)$/o then @plugin= s.reader('MyCommands', [$1])
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 when /^e (.*)$= /o then @plugins.reader('MyExtras',=A0=A0 $1.splitdata)
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 when /^s (.*)$/o then @plugins.reader('MySynonyms= 9;, $1.splitdata)
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 when /^r (.*)$/o then @plugins.read= er('MyRegisters',$1.splitdata)
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 when /^p (.*)$= /o then @plugins.reader('MyPlugins',=A0 $1.splitdata)
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 when /^x (.*)$/o then @plugins.reader('MyKeys',= =A0=A0=A0=A0 $1.splitdata)
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 when /^r (.*)$/o then # no= thing, not handled here
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 else
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0 # report("unknown entry #{line[0,1]} in line #{line= .chomp}")
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 end
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0=A0=A0=A0=A0 e= nd
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= f.close
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 end =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 else
=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 report("unable to locate #{tuifil= e}")
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 end
=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 rescue
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 report("fatal error in parsing #{tuifile}")
=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 @filename =3D 'texutil'
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 else
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0 @filename =3D filename
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 en= d
=A0=A0=A0=A0=A0=A0=A0 end
=A0=A0=A0
---------------------------= ------------------------------------=A0=A0=A0

From the debugging li= nes that are expelled, it is clear that the line in the .tui file that trig= gers the problem is:

=A0=A0=A0 c \thisisbytesequence{ ...non-ASCII codes... }

and, pr= ecisely, it s the second line of the 'case':

=A0=A0=A0 when = /^c (.*)$/o then @plugins.reader('MyCommands', [$1])

which p= rocesses the .tui line and triggers the 'rescue' clause. So I think= the problem lies in the "digestion"
of non-ASCII characters by the last version of Ruby.

I don't kn= ow what is the meaning of the \thisisbytesequence line in ConTeXt
and t= he maening of those non-ASCII chars. I followed the @plugins.reader('My= Commands', [$1])
and figured out that what raises the exception happens before the @plugins.= reader method,
since it is never reached when the=A0 "\thisisbytese= quence" line is processed.

To finish, I think this can be of in= terest for Ruby guys, but is also
breaking ConTeXt for anyone with ruby 1.9.1 installed in Windows.

Th= anks,

Jose Soares Augusto

jasaugusto@gmail.com
ja= sa@fisica.fc.ul.pt
--0016364166ffdcfdf2046e99cb20-- --===============1920186617== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___________________________________________________________________________________ --===============1920186617==--