From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.comp.tex.context/52161 Path: news.gmane.org!not-for-mail From: Jose Augusto Newsgroups: gmane.comp.tex.context Subject: Re: Ruby 1.9.1 and non-ascii char parsing in .tui file Date: Mon, 10 Aug 2009 16:15:06 +0100 Message-ID: <1781d7210908100815w251c33cj96aabfc9c0815ea5@mail.gmail.com> References: <1781d7210908081216n46c62018kb3a5f14643afb396@mail.gmail.com> <4A7F2A0D.2010907@wxs.nl> <1781d7210908092015n2d57ae21n6f09fd66729e41ac@mail.gmail.com> Reply-To: mailing list for ConTeXt users NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1869060575==" X-Trace: ger.gmane.org 1249917335 2228 80.91.229.12 (10 Aug 2009 15:15:35 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 10 Aug 2009 15:15:35 +0000 (UTC) To: mailing list for ConTeXt users Original-X-From: ntg-context-bounces@ntg.nl Mon Aug 10 17:15:27 2009 Return-path: Envelope-to: gctc-ntg-context-518@m.gmane.org Original-Received: from [195.12.62.10] (helo=balder.ntg.nl) by lo.gmane.org with esmtp (Exim 4.50) id 1MaWat-0000rv-62 for gctc-ntg-context-518@m.gmane.org; Mon, 10 Aug 2009 17:15:27 +0200 Original-Received: from localhost (localhost [127.0.0.1]) by balder.ntg.nl (Postfix) with ESMTP id A312BC9A86; Mon, 10 Aug 2009 17:15:14 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at balder.ntg.nl Original-Received: from balder.ntg.nl ([127.0.0.1]) by localhost (balder.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id jN-SIr8z6A9d; Mon, 10 Aug 2009 17:15:12 +0200 (CEST) Original-Received: from balder.ntg.nl (localhost [127.0.0.1]) by balder.ntg.nl (Postfix) with ESMTP id 116BCC9A77; Mon, 10 Aug 2009 17:15:12 +0200 (CEST) Original-Received: from localhost (localhost [127.0.0.1]) by balder.ntg.nl (Postfix) with ESMTP id 91E59C9A77 for ; Mon, 10 Aug 2009 17:15:10 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at balder.ntg.nl Original-Received: from balder.ntg.nl ([127.0.0.1]) by localhost (balder.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id W+wdjZPgIg+J for ; Mon, 10 Aug 2009 17:15:08 +0200 (CEST) Original-Received: from mail-bw0-f207.google.com (mail-bw0-f207.google.com [209.85.218.207]) by balder.ntg.nl (Postfix) with ESMTP id 04C66C9A68 for ; Mon, 10 Aug 2009 17:15:07 +0200 (CEST) Original-Received: by bwz3 with SMTP id 3so2661553bwz.2 for ; Mon, 10 Aug 2009 08:15:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=wJIxSm2MKMukuIvfRori6uNRyl9XKcpABfpOdfmteKQ=; b=Y1Fdc6Fr7WfGXT2iq7xv8ZP/gcrwU9LRCZnKlMuzgUJ4InyeFWUlnEhrNZ9NBT9YCY Rqwma+Bp6CftuYGc9byFeuOjvbA/wlTkyIMPmIZ/waT75zEdZQqH294V+E4825/0vS7o Q536veoiDMqStW3Y92JifPirBKjQCnZG/YtKU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=xSYmxLEBb/Rj+Qi8cdDrba7nO/FOG/YjyyyQYDrPJLY16gb6Dfsp22001eGM8Rt7qP JZtmKomLVNfqMUM72M0pMwvUmyoeO4Zw+uXuHWRilgLBKQNHQsFwobXpX42oVbGIkogU 8fcFtFMGvf6IiXd66X6pYMXPTag+fdS8TgDiQ= Original-Received: by 10.223.124.17 with SMTP id s17mr119225far.67.1249917307110; Mon, 10 Aug 2009 08:15:07 -0700 (PDT) In-Reply-To: <1781d7210908092015n2d57ae21n6f09fd66729e41ac@mail.gmail.com> X-BeenThere: ntg-context@ntg.nl X-Mailman-Version: 2.1.12 Precedence: list List-Id: mailing list for ConTeXt users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: ntg-context-bounces@ntg.nl Errors-To: ntg-context-bounces@ntg.nl Xref: news.gmane.org gmane.comp.tex.context:52161 Archived-At: --===============1869060575== Content-Type: multipart/alternative; boundary=001636c5adfe16c5640470cb0d29 --001636c5adfe16c5640470cb0d29 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi all, I think I solved the problem. At least for my actual errors... I read the following net article about string coding in ruby 1.9 and up: http://blog.grayproductions.net/articles/ruby_19s_string With that info at hand, I made two >> brute-force "trial" patches << (read the above article to see why I call them "brute force" :-) ) in two of the ruby context files where problems were arising (original line numbers are shown): ### .../scripts/context/ruby/base/tex.rb ## (946) case str.chomp ===>>> str = str.force_encoding("ISO-8859-1") if RUBY_VERSION >= "1.9" case str.chomp ### .../scripts/context/ruby/base/texutil.rb ### (1033) case line.chomp ===>>> line = line.force_encoding("ISO-8859-1") if RUBY_VERSION >= "1.9" case line.chomp The error is due to the fact that, by default, ruby 1.9 considers strings as US-ASCII and complains when finding chars not in that encoding. I don't know how to solve the problem for people writing in other encoding which is not ISO-8859-1. I tried the above with UTF-8 instead of ISO-8859-1 and it didn't work. Finally I don't know if there are any other places (at least case expressions) in the context ruby scripts where the problem might also show up. Kind regards, J. Augusto On Mon, Aug 10, 2009 at 4:15 AM, Jose Augusto wrote: > Hi all, > > Ok, here it goes. Atached are the files used in the test. > > The problem as reported in the >> previous email << used > the file with the offending chars wrapped in a main file, which was just: > > \starttext > \input zzz.tex > \stoptext > > That is, the offending chars were in zzz.tex. > > In that example I noticed the error because the cross-refs > in the equation numbering were not working. > The parsing of the .tui file by ruby 1.9.1 failed. Then I saw the errors. > > But then I made >> a single context file <<, which goes attached > with the correct results (tui, tuo, pdf), obtained with ruby 1.8.7. > > Howver, when i run ruby 1.9.1 with patch 129 (the last one), in this > single tex file (attached) now the first pass don't work! > Here is the result (in windows xp), which proves ruby 1.9.1 doesn't like > the non US-ASCII chars :-) > > F:\ANOS\ano09-10-pen\NotasProcSinal>texexec test1.tex > TeXExec | processing document 'test1.tex' > TeXExec | no ctx file found > D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:946:in `===': > invalid byte sequence in US-ASCII (ArgumentError) > from > D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:946: in > `scantexcontent' > from > D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:1907: in > `processfile' > from > D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:1143: in > `block (2 levels) in processtex' > from > D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:1133: in > `timedrun' > from > D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:1142: in > `block in processtex' > from > D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:1139: in > `each' > from > D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:1139: in > `processtex' > from > D:/Context/tex/texmf-context/scripts/context/ruby/texexec.rb:63:in`process' > from > D:/Context/tex/texmf-context/scripts/context/ruby/texexec.rb:53:in `main' > from > D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/switch.rb:133:in > `execute' > from > D:/Context/tex/texmf-context/scripts/context/ruby/texexec.rb:787:in `
' > > Thanks all for your interest, > > Kind regards > > J. Augusto > > > On Sun, Aug 9, 2009 at 8:57 PM, Hans Hagen wrote: > >> Jose Augusto wrote: >> >> when /^c (.*)$/o then @plugins.reader('MyCommands', [$1]) >>> >> >> what if you remove the o (/o) >> >> can you find out what changed between 1.8 and 1.9 ... actually 1.9 is the >> stepping stone to 2.0 and 2 versions can be incompatible to 1 versions >> >> also, can you make a test file so that we can see if there's a platform >> dependency? >> >> >> ----------------------------------------------------------------- >> Hans Hagen | PRAGMA ADE >> Ridderstraat 27 | 8061 GH Hasselt | The Netherlands >> tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com >> | www.pragma-pod.nl >> ----------------------------------------------------------------- >> >> ___________________________________________________________________________________ >> If your question is of interest to others as well, please add an entry to >> the Wiki! >> >> maillist : ntg-context@ntg.nl / >> http://www.ntg.nl/mailman/listinfo/ntg-context >> webpage : http://www.pragma-ade.nl / http://tex.aanhet.net >> archive : https://foundry.supelec.fr/projects/contextrev/ >> wiki : http://contextgarden.net >> >> ___________________________________________________________________________________ >> > > --001636c5adfe16c5640470cb0d29 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi all,

I think I solved the problem. At least for my actual errors.= ..
I read the following net article about string coding in ruby 1.9 and = up:

=A0=A0=A0=A0=A0 http://blog.grayproductions.net/articles/ruby_19s_stri= ng

With that info at hand, I made two >> brute-force "trial&quo= t; patches << (read the
above article to see why I call them &quo= t;brute force" :-) ) in two of the ruby context files
where problem= s were arising (original line numbers are shown):

###=A0 .../scripts/context/ruby/base/tex.rb=A0 ##
(946)=A0 case str.= chomp=A0=A0=A0=A0
=A0=A0=A0 =3D=3D=3D>>> =A0=A0=A0=A0
=A0 = =A0 =A0=A0=A0 str =3D str.force_encoding("ISO-8859-1") if RUBY_VE= RSION >=3D "1.9"
=A0=A0=A0 =A0=A0=A0 case str.chomp

### .../scripts/context/ruby/base/texutil.rb ###
(1033)=A0 case line= .chomp=A0
=A0=A0 =3D=3D=3D>>>=A0=A0=A0=A0=A0 =A0=A0=A0 =A0=A0= =A0
=A0=A0=A0 =A0=A0 line =3D line.force_encoding("ISO-8859-1"= ;) if RUBY_VERSION >=3D "1.9"
=A0=A0=A0 =A0=A0 case line.ch= omp

The error is due to the fact that, by default, ruby 1.9 considers strin= gs as
US-ASCII and complains when finding chars not in that encoding.
I don't know how to solve the problem for people writing in other =
encoding which is not ISO-8859-1. I tried the above with UTF-8
instead = of ISO-8859-1 and it didn't work.

Finally I don't know if th= ere are any other places (at least case expressions) in the
context rub= y scripts where the problem might also show up.

Kind regards,

J. Augusto


O= n Mon, Aug 10, 2009 at 4:15 AM, Jose Augusto <jasaugusto@gmail.com> wrote:=
Hi all,

Ok= , here it goes. Atached are the files used in the test.

The problem = as reported in the >> previous email << used
the file with the offending chars wrapped in a main file, which was just:
\starttext
\input zzz.tex
\stoptext

That is, the offending= chars were in zzz.tex.

In that example I noticed the error because = the cross-refs
in the equation numbering were not working.
The parsin= g of the .tui file by ruby 1.9.1 failed. Then I saw the errors.

But then I made >> a single context file <<, which goes att= ached
with the correct results (tui, tuo, pdf), obtained with ruby 1.8.7= .
=A0
Howver, when i run ruby 1.9.1 with patch 129 (the last one), i= n this
single tex file (attached) now the first pass don't work!
Here is t= he result (in windows xp), which proves ruby 1.9.1 doesn't like
the= non US-ASCII chars :-)

F:\ANOS\ano09-10-pen\NotasProcSinal>texex= ec test1.tex
TeXExec | processing document 'test1.tex'
TeXExec | no ctx file = found
D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:946:= in `=3D=3D=3D': invalid byte sequence in US-ASCII (ArgumentError)
= =A0=A0=A0=A0=A0=A0=A0 from D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/rub= y/base/tex.rb:946: in `scantexcontent'
=A0=A0=A0=A0=A0=A0=A0 from D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/rub= y/base/tex.rb:1907: in `processfile'
=A0=A0=A0=A0=A0=A0=A0 from D:/C= ontext/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:1143: in `block (= 2 levels) in processtex'
=A0=A0=A0=A0=A0=A0=A0 from D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/rub= y/base/tex.rb:1133: in `timedrun'
=A0=A0=A0=A0=A0=A0=A0 from D:/Cont= ext/tex/texmf-context/SCRIPTS/CONTEXT/ruby/base/tex.rb:1142: in `block in p= rocesstex'
=A0=A0=A0=A0=A0=A0=A0 from D:/Context/tex/texmf-context/S= CRIPTS/CONTEXT/ruby/base/tex.rb:1139: in `each'
=A0=A0=A0=A0=A0=A0=A0 from D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/rub= y/base/tex.rb:1139: in `processtex'
=A0=A0=A0=A0=A0=A0=A0 from D:/Co= ntext/tex/texmf-context/scripts/context/ruby/texexec.rb:63:in`process'<= br>=A0=A0=A0=A0=A0=A0=A0 from D:/Context/tex/texmf-context/scripts/context/= ruby/texexec.rb:53:in `main'
=A0=A0=A0=A0=A0=A0=A0 from D:/Context/tex/texmf-context/SCRIPTS/CONTEXT/rub= y/base/switch.rb:133:in `execute'
=A0=A0=A0=A0=A0=A0=A0 from D:/Cont= ext/tex/texmf-context/scripts/context/ruby/texexec.rb:787:in `<main>&= #39;

Thanks all for your interest,

Kind regards

J. Augusto

On Sun, Aug 9, 2009 at 8:57 PM, Hans Hagen <pragma= @wxs.nl> wrote:
Jose Augusto wrote:

=A0 =A0 =A0 =A0when /^c (.*)$/o then @plugins.reader('MyCommands',= [$1])

what if you remove the o (/o)

can you find out what changed between 1.8 and 1.9 ... actually 1.9 is the s= tepping stone to 2.0 and 2 versions can be incompatible to 1 versions

also, can you make a test file so that we can see if there's a platform= dependency?


-----------------------------------------------------------------
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0Hans Hagen | PRAGMA ADE
=A0 =A0 =A0 =A0 =A0 =A0 =A0Ridderstraat 27 | 8061 GH Hasselt | The Netherl= ands
=A0 =A0 tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 | ww= w.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________= ________
If your question is of interest to others as well, please add an entry to t= he Wiki!

maillist : ntg-cont= ext@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage =A0: http://= www.pragma-ade.nl / http://tex.aanhet.net
archive =A0: https://foundry.supelec.fr/projects/contextrev/
wiki =A0 =A0 : http:= //contextgarden.net
___________________________________________________________________________= ________


--001636c5adfe16c5640470cb0d29-- --===============1869060575== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___________________________________________________________________________________ --===============1869060575==--