[m-database] How to properly add module arguments?

[m-database] How to properly add module arguments?

[Paul Mazaitis via ntg-context]

Dear list,

## The Incredibly Short Version

I am trying to extend m-database, and I want to add a flag to control behavior. I can get it to work, but I am having trouble understanding how to do it properly.

## The Short Version

I have a project where I wish to incorporate many CSV files as Natural Tables.

I want to keep the headers in the CSV files so they're useful in other contexts.

The `m-database` module doesn't have a feature to suppress the header row, from what I can see.

(I know `handlecsv` can do this, but I'm having troubles with that; if the `m-database` line of inquiry doesn't work out, I'll go back it, but I figure I'll focus on this solution for now.)

Looking at the module sources, I can implement the functionality I need in the Lua part of the module, but:

I seem to only be able to bodge in additional arguments for the commands in the module.

On first try, I got the error: `tex error       > tex error on line 30 in file /Users/pmazaitis/Opt/context/tex/texmf-context/tex/context/modules/mkiv/m-databaseplus.mkiv: ! Undefined control sequence`

And then: `The control sequence at the end of the top line of your error message was never
\def'ed.`

Looking at the docs points me to a solution using  `\def`, but it's not clear to me how that's working in the case of the database module (the database module seems to be initializing these values someplace else?).

If I \def the settings variable in m-database.mkiv,  I can get it to work, but I don't think that's the correct way to go about this.

Question: how do I properly define new module arguments for the database module?

## The Long Version, With MWE

### Files Associated With the MWE

Apologies for all of the file inclusions, but I want to be complete.


Here's the .csv file I'm using for the MWE:

--- begin database-mwe.csv ---

Ref. No.,Part No.,Description,Remarks
,RD14C82E000J,Carbon 000𝝮 ±5% 1/4W,

---  end database-mwe.csv  ---

Some notes on database-mwe.csv:

I'd like to keep the header row in place to make this file useful for other tools.

These are lists of electronic parts, so I'd like to include uppercase omega, lower case mu, and percent sign characters in the CSV file.


Here's the .tex file I'm using for the MWE:

--- begin database-mwe.tex ---

\usemodule[database]

\setupdatabase[quotechar={"},separator={,}]

\defineseparatedlist
  [PartsTable]
  [separator=comma,left=\bTD,right=\eTD,first=\bTR,last=\eTR,before={
\bTABLE[split=repeat,option=stretch]
  \bTABLEhead
  \bTR
    \bTH Ref. No. \eTH \bTH Part No. \eTH \bTH Description \eTH \bTH Remarks \eTH
  \eTR
  \eTABLEhead
  \bTABLEbody
   },
   after={
  \eTABLEbody
  \eTABLE
   }]

\starttext

test

\startasciimode
\processdatabasefile[PartsTable][parts_list-test.csv]
\stopasciimode

\stoptext

---  end database-mwe.tex  ---

Some notes on database-mwe.tex:

This actually works pretty well with the distributed m-database module!

Some of these lists are quite long, so I want to use a TABLEhead to replicate the headings at the top of a new page. I don't think there's a way to automate this with `m-database`, so I'm okay with hard-coding the headers. This does mean I get a double header row at the top of the table. I want to suppress the header row; I'd like to implement this behavior in the module behind a switch.

(I'm using the `asciimode` environment to dodge problems with unescaped `%` characters and Greek letters. I tried adding the startasciimode/stopasciimode commands to various keys in the \defineseparatedlist command, but I didn't get anywhere with it - I got errors. A problem for another time...)

I think I've got a good handle on how to do this in Lua, but I'm struggling with what to add to the switch appropriately.

In attempting to add a switch, I tried the following.

Based off of https://wiki.contextgarden.net/Modules#Modules_writing_guidelines,
I added a stanza to the appropriate interface file:

--- begin stanza added to i-database.xml ---

                <cd:parameter name="skipheader">
                    <cd:constant type="yes"/>
                    <cd:constant type="no" default="yes"/>
                </cd:parameter>

---  end stanza added to i-database.xml  ---

I modeled that after the stanza for the argument `strip`. I think that's the only thing I need to do in the interface file.

Changing the i-database.xml file does not break the build.

However, as soon as I add the skipheader argument to the m-database.mkiv file:

--- begin definition in m-database.mkiv ---

\setupdatabase
  [\c!separator={,},
   \c!quotechar=,
   \c!commentchar=,
   \c!strip=\v!no,
   \c!skipheader=\v!no, % <- Added, following the pattern with strip
   \c!before=,
   \c!after=,
   \c!first=,
   \c!last=,
   \c!left=,
   \c!right=]

---  end definition in m-database.mkiv  ---


...the build fails (with the error from the Short Version section).

If I then add the following \def to m-database.mkiv before the \setupdatabase command:

    \def\c!skipheader {skipheader}

...the build succeeds! I can now pass that settings value to m-database.lua and use it to modify the processing logic.

My question: how do I \emph{properly} define new module arguments for the database module?

Or: is adding extra \def commands the way to do it?

Many, many thanks!

    -Paul





___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: [m-database] How to properly add module arguments?

[Max Chernoff via ntg-context]

Hi Paul,

> I added a stanza to the appropriate interface file:
> 
> --- begin stanza added to i-database.xml ---
> 
> I modeled that after the stanza for the argument `strip`. I think
> that's the only thing I need to do in the interface file.

I'd recommend completely ignoring the interface i-*.xml files. They
don't do anything particularly useful for a third-party module.

>    \c!skipheader=\v!no, % <- Added, following the pattern with strip

This line is your issue. Inside \unprotect...\protect, "!" is treated as
a letter, so you're trying to expand the value of the macro
"\c!skipheader". The problem is that that macro isn't defined anywhere. 

Backing up a little further, the \c!... commands are all string
*C*onstants. If you see a command with a name like \c!blah, it almost
certain expands to exactly the string "blah". Why is this useful? Well,
if you run ConTeXt in another language, then the \c!... commands will
instead contain that string but in the other language. So something like
\c!left would be "left" in English, "gauche" in French, "links" in
German, "vlevo" in Czech, etc. 

You could define \c!skipheader to just be "skipheader", or you could
give it a translated value for 10 or so languages. But the easier
solution is to just use the string "skipheader" directly:

   \setupdatabase
     [\c!separator={,},
      \c!quotechar=,
      \c!commentchar=,
      \c!strip=\v!no,
      skipheader=\v!no, % <-- HERE
      \c!before=,
      \c!after=,
      \c!first=,
      \c!last=,
      \c!left=,
      \c!right=]
   
As a more general suggestion, it is usually a really bad idea to modify the
standard ConTeXt files since any changes will be overwritten on updates.
Probably the best solution is to make a new file called "t-xdatabase.mkiv"
stored in your texmf-local/, start the file with "\usemodule[database]"
then add any changes after that. Or, just copy and paste the original file
into your new file. (Maybe you were doing this already though)
   
Hope this helps,
-- Max
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________

Re: [m-database] How to properly add module arguments?

[Paul Mazaitis via ntg-context]

On 26 Feb 2023, at 5:36, Max Chernoff via ntg-context wrote:

> I'd recommend completely ignoring the interface i-*.xml files. They
> don't do anything particularly useful for a third-party module.

It's good to know there's no dependency there (but I do like to keep things updated :) ).

> This line is your issue. Inside \unprotect...\protect, "!" is treated as
> a letter, so you're trying to expand the value of the macro
> "\c!skipheader". The problem is that that macro isn't defined anywhere.

A ha! Many thanks; I've gotten it working, along with this hint:

>       skipheader=\v!no, % <-- HERE

...so I think I'm okay.

(I don't mind doing the translations, but that's probably outside the scope of this project.)

> Backing up a little further, the \c!... commands are all string
> *C*onstants. If you see a command with a name like \c!blah, it almost
> certain expands to exactly the string "blah". Why is this useful? Well,
> if you run ConTeXt in another language, then the \c!... commands will
> instead contain that string but in the other language. So something like
> \c!left would be "left" in English, "gauche" in French, "links" in
> German, "vlevo" in Czech, etc.

This, too, was useful - as per the Scratch Variables page in the System Macros section of the garden (https://wiki.contextgarden.net/System_Macros/Scratch_Variables) I've updated my tree-sitter parser (https://github.com/pmazaitis/tree-sitter-context_en) to put these in the parse tree; I don't know how useful that is, but it seems to me that being able to call these out might help.

> As a more general suggestion, it is usually a really bad idea to modify the
> standard ConTeXt files since any changes will be overwritten on updates.
> Probably the best solution is to make a new file called "t-xdatabase.mkiv"
> stored in your texmf-local/, start the file with "\usemodule[database]"
> then add any changes after that. Or, just copy and paste the original file
> into your new file. (Maybe you were doing this already though)

Oh! Indeed: my working ConTeXt environment is under version control;  experimental nonsense like this gets safely tucked away in a branch (my thinking is that I want to try to change as few new variables as possible - still learning!).

> Hope this helps,

Absolutely; many thanks!

> -- Max

	-Paul

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : https://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : https://contextgarden.net
___________________________________________________________________________________