substitutions

substitutions

[Idris Samawi Hamid ادريس سماوي حامد]

Dear gang,

Sometimes I use the following for simple substitutions:

\defineactivecharacter  '  {\otfchar{quoteright}}

But is there a more general mechanism to do things

\definesubstitution{<string1>}{<string2>}

eg

\definesubstitution{--}{–}

Of course ConTeXt already provides this particular substitution but I'm  
interested in such a mechanism for more general purposes.

Best wishes
Idris

-- 
Professor Idris Samawi Hamid, Editor-in-Chief
International Journal of Shi`i Studies
Department of Philosophy
Colorado State University
Fort Collins, CO 80523
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions

[Mojca Miklavec]

On Mon, Oct 6, 2008 at 1:57 AM, Idris Samawi Hamid ادريس سماوي حامد wrote:
> Dear gang,
>
> Sometimes I use the following for simple substitutions:
>
> \defineactivecharacter  '  {\otfchar{quoteright}}
>
> But is there a more general mechanism to do things
>
> \definesubstitution{<string1>}{<string2>}
>
> eg
>
> \definesubstitution{--}{–}
>
> Of course ConTeXt already provides this particular substitution

In mkii it doesn't. Fonts do that substitution. In mkiv a hack is
applied to fonts (not to TeX macros), so that this particular
substitution works.

> but I'm
> interested in such a mechanism for more general purposes.

In mkii this is not possible, except with some ugly hacks. You can
easily substitute a single character by making it active. To
substitute whole words, you either need to modify fonts, or write some
dirty macros.

In mkiv you can either apply some "patches" to fonts (search for tlig
in ConTeXt source, for example font-otf.lua, also, there are some fea
files in fonts/fea/context capable of doing that), or change input
text while reading/digesting some TeX file.

Hans and Taco can probably tell you more.

Mojca
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions

[Hans Hagen]

Mojca Miklavec wrote:
> On Mon, Oct 6, 2008 at 1:57 AM, Idris Samawi Hamid ادريس سماوي حامد wrote:
>> Dear gang,
>>
>> Sometimes I use the following for simple substitutions:
>>
>> \defineactivecharacter  '  {\otfchar{quoteright}}
>>
>> But is there a more general mechanism to do things
>>
>> \definesubstitution{<string1>}{<string2>}
>>
>> eg
>>
>> \definesubstitution{--}{–}
>>
>> Of course ConTeXt already provides this particular substitution
> 
> In mkii it doesn't. Fonts do that substitution. In mkiv a hack is
> applied to fonts (not to TeX macros), so that this particular
> substitution works.
> 
>> but I'm
>> interested in such a mechanism for more general purposes.
> 
> In mkii this is not possible, except with some ugly hacks. You can
> easily substitute a single character by making it active. To
> substitute whole words, you either need to modify fonts, or write some
> dirty macros.
> 
> In mkiv you can either apply some "patches" to fonts (search for tlig
> in ConTeXt source, for example font-otf.lua, also, there are some fea
> files in fonts/fea/context capable of doing that), or change input
> text while reading/digesting some TeX file.
> 
> Hans and Taco can probably tell you more.

it all depends on what we want to achieve ...

- we can have font fixers (dynamically extend fonts with features, but 
this demands knowledge of the font)

- we can have replacements in the node list (attribute driven, 
relatively easy to implement)

so before i start looking into this, we need to investigate what is 
needed (and what for)

(only mkiv)

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions

[Idris Samawi Hamid ادريس سماوي حامد]

On Tue, 07 Oct 2008 04:53:40 -0600, Mojca Miklavec  
<mojca.miklavec.lists@gmail.com> wrote:

> On Mon, Oct 6, 2008 at 1:57 AM, Idris Samawi Hamid ادريس سماوي حامد

>> Sometimes I use the following for simple substitutions:
>>
>> \defineactivecharacter  '  {\otfchar{quoteright}}
>>
>> But is there a more general mechanism to do things
>>
>> \definesubstitution{<string1>}{<string2>}
>>
>> eg
>>
>> \definesubstitution{--}{–}
>>
>> Of course ConTeXt already provides this particular substitution
>
> In mkii it doesn't. Fonts do that substitution. In mkiv a hack is
> applied to fonts (not to TeX macros), so that this particular
> substitution works.

Hmm, so this is much more complicated than I thought...

>> but I'm
>> interested in such a mechanism for more general purposes.
>
> In mkii this is not possible, except with some ugly hacks. You can
> easily substitute a single character by making it active. To
> substitute whole words, you either need to modify fonts, or write some
> dirty macros.

I see. In any case, mkii is "obsolete" for my purposes, if/when I get to  
writing a book on ConTeXt it will cover mkiv only.

> In mkiv you can either apply some "patches" to fonts (search for tlig
> in ConTeXt source, for example font-otf.lua, also, there are some fea
> files in fonts/fea/context capable of doing that), or change input
> text while reading/digesting some TeX file.

tlig was buggy, but it works in the latest beta. Thank you, Hans!

> Hans and Taco can probably tell you more.

And Thank you, Mojca!

Idris

-- 
Professor Idris Samawi Hamid, Editor-in-Chief
International Journal of Shi`i Studies
Department of Philosophy
Colorado State University
Fort Collins, CO 80523
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions

[Idris Samawi Hamid ادريس سماوي حامد]

On Tue, 07 Oct 2008 12:16:42 -0600, Hans Hagen <pragma@wxs.nl> wrote:

>> Hans and Taco can probably tell you more.
> it all depends on what we want to achieve ...

> - we can have font fixers (dynamically extend fonts with features, but
> this demands knowledge of the font)
> - we can have replacements in the node list (attribute driven,
> relatively easy to implement)
> so before i start looking into this, we need to investigate what is
> needed (and what for)

For things like my journal, I get lots of inconsistent transliterations,  
and we are not yet at the stage where we can enforce behavior on authors.  
I would like to be able to do things like

Mohamed -> Muḥammad

which would save me a lot of time in proofreading, editing, etc. Getting  
the transliteration for the entire journal consistent is a tedious pain,  
and eventually I can have a nice database that will cover most cases. This  
is also much more convenient than using control-sequence definitions and  
watching out for spaces etc.

> (only mkiv)

Indeed ;-)

Best wishes
Idris

-- 
Professor Idris Samawi Hamid, Editor-in-Chief
International Journal of Shi`i Studies
Department of Philosophy
Colorado State University
Fort Collins, CO 80523
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions

[Mojca Miklavec]

On Wed, Oct 8, 2008 at 2:41 PM, Idris Samawi Hamid ادريس سماوي حامد wrote:
>
> I would like to be able to do things like
>
> Mohamed -> Muḥammad
>
> which would save me a lot of time in proofreading, editing, etc.

It's easy to write a simple ruby/perl(or even lua) script that will do
such changes for you *before* typesetting articles. That's probably
much better, since you also want to have clean sources at some point.

Of course, writing the list of substitutions still needs to be done by
you, but that's in either case.

If you need that, write kind-of-specification how you would like to
use such a command.

Mojca
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions

[Idris Samawi Hamid]

On Wed, 08 Oct 2008 08:46:21 -0600, Mojca Miklavec  
<mojca.miklavec.lists@gmail.com> wrote:

> On Wed, Oct 8, 2008 at 2:41 PM, Idris Samawi Hamid ادريس سماوي حامد  
> wrote:
>>
>> I would like to be able to do things like
>>
>> Mohamed -> Muḥammad
>>
>> which would save me a lot of time in proofreading, editing, etc.
>
> It's easy to write a simple ruby/perl(or even lua) script that will do
> such changes for you *before* typesetting articles. That's probably
> much better, since you also want to have clean sources at some point.

Yes, but the above is just one kind of example. Indeed, what I am thinking  
of is just an extension of the kind of configurability texies like to do.  
For your average user -- or a secretary -- we don't want to have to point  
them to lua or ruby etc...

> Of course, writing the list of substitutions still needs to be done by
> you, but that's in either case.
>
> If you need that, write kind-of-specification how you would like to
> use such a command.

Maybe I misunderstand... I could see something like I mentioned in the  
original mail

\definesubstitution{<string1>}{<string2>}

or even a database approach:

\definesubstitutions[
<string1>,<string1'>|
<string2>,<string2'>|
<string3>,<string3'>
]

Here is another example. I generally prefer an en-dash padded with spaces  
-- and with a no-break space after -- to an em-dash for parenthetical  
constructions. So I would like to do, eg

\definesubstitution{<emdash>}{ <endash>~}

Again, this saves time in extensive editing, and I can stay in TeX without  
fiddling so much with scripting, CTRL-F/H/R etc. It maintains control in  
case I change my mind and want to revert etc. For long documents and  
projects this gets unwieldy pretty fast and I end up with lots of typos  
etc.

I guess we could get really funky and add a regular expressions engine  
using lua:

\definesubstitution[regex]{<string1>}{<string2>}

And I'm sure Hans, Taco, and other programmers can think of other nifty  
macros...

But the basic functionality would suffice for now :-)

Best wishes
Idris

-- 
Professor Idris Samawi Hamid, Editor-in-Chief
International Journal of Shi`i Studies
Department of Philosophy
Colorado State University
Fort Collins, CO 80523

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions

[Mojca Miklavec]

On Wed, Oct 8, 2008 at 2:41 PM, Idris Samawi Hamid ادريس سماوي حامد wrote:
>
> I would like to be able to do things like
>
> Mohamed -> Muḥammad

Before you get the mkiv code you can start collecting wrong words :)

Mojca


usage: ruby scriptname.rb < input.tex > output.tex

#!/usr/bin/env ruby

subs_hash = {
  "Mohamed" => "Muḥammad",
  "Idris" => "ادريس",
}

while line = STDIN.gets
  subs_hash.each do |key,value|
    line.gsub!(Regexp.new(key), value)
  end
  puts line
end
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions

[Khaled Hosny]

[-- Attachment #1.1: Type: text/plain, Size: 1172 bytes --]

(I know this an old thread, but I just happen to exactly the same
feature Idris was asking for)

On Thu, Oct 09, 2008 at 12:53:47AM +0200, Mojca Miklavec wrote:
> usage: ruby scriptname.rb < input.tex > output.tex
> 
> #!/usr/bin/env ruby
> 
> subs_hash = {
>   "Mohamed" => "Muḥammad",
>   "Idris" => "ادريس",
> }
> 
> while line = STDIN.gets
>   subs_hash.each do |key,value|
>     line.gsub!(Regexp.new(key), value)
>   end
>   puts line
> end

Such approach has a major disadvantage, one have to take care of not
altering macros and tex commands or we may end with a broken document (I
have more general use case where the to be substituted strings can
occur in macros as well), and we know that "only TeX can read TeX" :)

I'm thinking in lua callback that get fed with text buffers in a similar
way to OTPs, so one can use regular expressions to pre-process the text
without messing with node list, think in code that replaces 2nd with
2\high{nd} etc. which would be very complex to be done at node list
level.

Regards,
 Khaled


-- 
 Khaled Hosny
 Arabic localiser and member of Arabeyes.org team
 Free font developer

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 487 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions: can i send this?

[Idris Samawi Hamid ادريس سماوي حامد]

On Fri, 27 Mar 2009 06:05:03 -0600, Khaled Hosny <khaledhosny@eglug.org>
wrote:

> (I know this an old thread, but I just happen to exactly the same
> feature Idris was asking for)
>
> On Thu, Oct 09, 2008 at 12:53:47AM +0200, Mojca Miklavec wrote:
>> usage: ruby scriptname.rb < input.tex > output.tex
>>
>> #!/usr/bin/env ruby
>>
>> subs_hash = {
>>   "Mohamed" => "Muḥammad",
>>   "Idris" => "ادريس",
>> }
>>
>> while line = STDIN.gets
>>   subs_hash.each do |key,value|
>>     line.gsub!(Regexp.new(key), value)
>>   end
>>   puts line
>> end
>
> Such approach has a major disadvantage, one have to take care of not
> altering macros and tex commands or we may end with a broken document (I
> have more general use case where the to be substituted strings can
> occur in macros as well), and we know that "only TeX can read TeX" :)
>
> I'm thinking in lua callback that get fed with text buffers in a similar
> way to OTPs, so one can use regular expressions to pre-process the text
> without messing with node list, think in code that replaces 2nd with
> 2\high{nd} etc. which would be very complex to be done at node list
> level.

If it's not so complex, have you considered overloading font features with
fea files? Now that Adobe has (finally!) completed a major overhaul of the
spec, including the features neeeded for arabic, that's an option.

But for your more complex needs, see the m-translate module. Maybe that
can be developed further? Note that Hans has pointed out that this is NOT
a core module and that it is preferable to fix the source beforehand.

سلام
Idris

-- 
Professor Idris Samawi Hamid, Editor-in-Chief
International Journal of Shi`i Studies
Department of Philosophy
Colorado State University
Fort Collins, CO 80523
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions: can i send this?

[Khaled Hosny]

[-- Attachment #1.1: Type: text/plain, Size: 1721 bytes --]

On Sat, Mar 28, 2009 at 07:27:07PM -0600, Idris Samawi Hamid ادريس سماوي حامد wrote:
> On Fri, 27 Mar 2009 06:05:03 -0600, Khaled Hosny <khaledhosny@eglug.org>
> wrote:
> If it's not so complex, have you considered overloading font features with
> fea files? Now that Adobe has (finally!) completed a major overhaul of the
> spec, including the features neeeded for arabic, that's an option.

Yes, it is an option, but not for complex operations as you know, and
also font dependant (not all fonts use Adobe glyph names etc.)

> But for your more complex needs, see the m-translate module. Maybe that
> can be developed further? Note that Hans has pointed out that this is NOT
> a core module and that it is preferable to fix the source beforehand.

That module shows exactly the problem I mentioned in my earlier message,
it processes the whole input file not only the text buffers that will be
actually typesetted, so If I've an \Idris macro it'll get replaced as
well, or I've to parse the file myself which isn't really an option.
OTP fits nicely in this, but it isn't as powerful as lua, and there seem
not to be much interest in maintaining it in the future.

I think, being able to process the to-be-typesetted text in lua is
desirable and have many use cases. Now we have callbacks for input
buffer, which is not very handy when you need to handle only the textual
material without messing with TeX commands and macros, and callbacks for
token and node list, which make very complex what is otherwise simple
task, like 1st -> 1\high{st} conversion, etc.

Regards,
 Khaled


-- 
 Khaled Hosny
 Arabic localiser and member of Arabeyes.org team
 Free font developer

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 487 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: substitutions: can i send this?

[Hans Hagen]

Khaled Hosny wrote:

> That module shows exactly the problem I mentioned in my earlier message,
> it processes the whole input file not only the text buffers that will be
> actually typesetted, so If I've an \Idris macro it'll get replaced as
> well, or I've to parse the file myself which isn't really an option.
> OTP fits nicely in this, but it isn't as powerful as lua, and there seem
> not to be much interest in maintaining it in the future.

we already have spellchecking (operates on node lists) so having some 
kind of replacement mechanism is doable too (not now, later when i've 
cleaned up some code)

otp's are gone from mkiv .. i simple don't enable the primitives any 
more and the filter mechanisms is not loaded; of course that coule 
become a module to be loaded at runtime if there is some real need for it

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________