support arguments on filters

support arguments on filters

[Aaron O'Leary]

Currently external filters implicitly take the name of the output
format as the first argument, but is there any reason not to allow
them to take more explicit arguments?

If there is way to do this already I can't work it out, beyond piping
directly to and from the filter:

    pandoc file.md --to json | myfilter --arg1 --arg2 | pandoc --from
json --to whatever


I'm thinking of a format like

    pandoc file.md --to whatever --filter='myfilter --arg1 --arg2'

but perhaps we'd want a separate --filter_args argument.

Re: support arguments on filters

[Jesse Rosenthal]

Aaron O'Leary <aaron.oleary-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> If there is way to do this already I can't work it out, beyond piping
> directly to and from the filter:

You can make filters aware of arbitrary document metadata, and then call

    pandoc file.md --filter=myfilter -M key1=val1 -M arg

This is what I do to offer variable features in filters. I'm not sure
if it fits your needs -- I tend to like it because it keeps the
metadata for filters nicely separated from the options to pandoc, and
also because you can specify the options in your YAML block.

Re: support arguments on filters

[BP Jonsson]

Den 2014-09-27 14:02, Jesse Rosenthal skrev:
> Aaron O'Leary <aaron.oleary-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
>> If there is way to do this already I can't work it out, beyond piping
>> directly to and from the filter:
>
> You can make filters aware of arbitrary document metadata, and then call
>
>      pandoc file.md --filter=myfilter -M key1=val1 -M arg
>
> This is what I do to offer variable features in filters. I'm not sure
> if it fits your needs -- I tend to like it because it keeps the
> metadata for filters nicely separated from the options to pandoc, and
> also because you can specify the options in your YAML block.
>

The only thing I miss is a way to give non-markdown values this 
way; you have to wrap them in backticks! :-(

What about an -O key=value option the args to which ends up 
unaltered in a _filter_options subtree of the metadata, or at 
least a convention that metadata keys which begin with an 
underscore have non-markdown values?

Re: support arguments on filters

[Matthew Pickering]

[-- Attachment #1.1: Type: text/plain, Size: 1283 bytes --]

I still think this would be nice to have for more complicated filters. 

On Saturday, September 27, 2014 at 12:13:59 PM UTC+1, Aaron O'leary wrote:
>
> Currently external filters implicitly take the name of the output 
> format as the first argument, but is there any reason not to allow 
> them to take more explicit arguments? 
>
> If there is way to do this already I can't work it out, beyond piping 
> directly to and from the filter: 
>
>     pandoc file.md --to json | myfilter --arg1 --arg2 | pandoc --from 
> json --to whatever 
>
>
> I'm thinking of a format like 
>
>     pandoc file.md --to whatever --filter='myfilter --arg1 --arg2' 
>
> but perhaps we'd want a separate --filter_args argument. 
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/52241fed-9d94-4dc9-8c48-0affd6325878%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 2659 bytes --]

Re: support arguments on filters

[John MacFarlane]

I suppose the reason filter arguments weren't allowed is
that pandoc treats the whole string (argument of --filter)
as the command.  Since pandoc just gets one string as
argument of --filter, it would need to do its own parsing
of this string into a command, options and arguments in
order for --filter='myfilter --arg1 --arg2' to work.
That's the main complication that I see, but it's
non-trivial.  Note that file paths can contain spaces and
hyphen characters.  I suppose we could split the string
at the first occurrence of '--', though, and let the first
part be the filter, the rest (split on spaces) the
arguments.


+++ Matthew Pickering [Nov 27 15 07:13 ]:
>   I still think this would be nice to have for more complicated filters.
>   On Saturday, September 27, 2014 at 12:13:59 PM UTC+1, Aaron O'leary
>   wrote:
>
>     Currently external filters implicitly take the name of the output
>     format as the first argument, but is there any reason not to allow
>     them to take more explicit arguments?
>     If there is way to do this already I can't work it out, beyond
>     piping
>     directly to and from the filter:
>         pandoc [1]file.md --to json | myfilter --arg1 --arg2 | pandoc
>     --from
>     json --to whatever
>     I'm thinking of a format like
>         pandoc [2]file.md --to whatever --filter='myfilter --arg1
>     --arg2'
>     but perhaps we'd want a separate --filter_args argument.
>
>   --
>   You received this message because you are subscribed to the Google
>   Groups "pandoc-discuss" group.
>   To unsubscribe from this group and stop receiving emails from it, send
>   an email to [3]pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>   To post to this group, send email to
>   [4]pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>   To view this discussion on the web visit
>   [5]https://groups.google.com/d/msgid/pandoc-discuss/52241fed-9d94-4dc9-
>   8c48-0affd6325878%40googlegroups.com.
>   For more options, visit [6]https://groups.google.com/d/optout.
>
>References
>
>   1. http://file.md/
>   2. http://file.md/
>   3. mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
>   4. mailto:pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
>   5. https://groups.google.com/d/msgid/pandoc-discuss/52241fed-9d94-4dc9-8c48-0affd6325878-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org?utm_medium=email&utm_source=footer
>   6. https://groups.google.com/d/optout

Re: support arguments on filters

[John MacFarlane]

+++ John MacFarlane [Nov 27 15 11:51 ]:
>I suppose the reason filter arguments weren't allowed is
>that pandoc treats the whole string (argument of --filter)
>as the command.  Since pandoc just gets one string as
>argument of --filter, it would need to do its own parsing
>of this string into a command, options and arguments in
>order for --filter='myfilter --arg1 --arg2' to work.
>That's the main complication that I see, but it's
>non-trivial.  Note that file paths can contain spaces and
>hyphen characters.  I suppose we could split the string
>at the first occurrence of '--', though, and let the first
>part be the filter, the rest (split on spaces) the
>arguments.

One more relevant thing.  Filters are passed the name of
the output format as the first argument.  So, if we allowed
options and arguments to be specified, we'd need to
rearrange things.  E.g.,

    pandoc -t latex --filter 'myfilter --option1 --option optarg'

would call the filter using

    myfilter latex --option1 --option optarg

This could be done but it's a bit ugly.

Re: support arguments on filters

[BP Jonsson]

What is really needed is metadata values which are not parsed as 
Markdown.  Since pandoc ignores fields ending in an underscore, 
what about leaving string values of fields beginning in an 
underscore, and their children, as raw text?

Alternatively since there already is a `--latex-engine-opt` what 
about a `--filter-option=KEY:VAL`/`-O KEY=VAL`? Rather than 
passing its fields separately to the filters it can be included in 
the metadata under a key `filter-options`. It would differ from 
the rest of the metadata by not being specially interpreted, just 
a mapping of keys to plain strings. It would of course be 
desirable that fields (and possibly hierarchical children) of that 
mapping can be set in a metadata block as well, with fields given 
on the command line overriding those set in metadata blocks on a 
per-field basis. Would that be reasonable?

FWIW I have written Perl code which I use in my filters which 
scans the metadata for keys which start with the filter name and 
walks the values of such fields extracting the first 'stringy' 
scalar value from each MetaInlines or MetaBlocks, returning a hash 
which mirrors the MetaMap and MetaList objects under those fields 
as plain hashes/arrays. The problem is that that code is often 
larger than the actual filter. I have turned it into a module, though.

/bpj

Den 2015-11-29 kl. 07:30, skrev John MacFarlane:
> +++ John MacFarlane [Nov 27 15 11:51 ]:
>> I suppose the reason filter arguments weren't allowed is
>> that pandoc treats the whole string (argument of --filter)
>> as the command.  Since pandoc just gets one string as
>> argument of --filter, it would need to do its own parsing
>> of this string into a command, options and arguments in
>> order for --filter='myfilter --arg1 --arg2' to work.
>> That's the main complication that I see, but it's
>> non-trivial.  Note that file paths can contain spaces and
>> hyphen characters.  I suppose we could split the string
>> at the first occurrence of '--', though, and let the first
>> part be the filter, the rest (split on spaces) the
>> arguments.
>
> One more relevant thing.  Filters are passed the name of
> the output format as the first argument.  So, if we allowed
> options and arguments to be specified, we'd need to
> rearrange things.  E.g.,
>
>     pandoc -t latex --filter 'myfilter --option1 --option optarg'
>
> would call the filter using
>
>     myfilter latex --option1 --option optarg
>
> This could be done but it's a bit ugly.
>

Re: support arguments on filters

[Matthew Pickering]

I think any discussion about metadata is a red herring. I don't want
to include this information in the document I am trying to process, I
want to put it in a makefile. Adding it as metadata is indirect and
annoying to have to deal with.

Perhaps it would be possible to use "shell" rather than "proc" in this section?

https://github.com/jgm/pandoc/blob/64b32e1e8194644df37a115200f929a9553c6d16/src/Text/Pandoc/Process.hs#L63

On Sun, Nov 29, 2015 at 8:39 PM, BP Jonsson <bpj-J3H7GcXPSITLoDKTGw+V6w@public.gmane.org> wrote:
> What is really needed is metadata values which are not parsed as Markdown.
> Since pandoc ignores fields ending in an underscore, what about leaving
> string values of fields beginning in an underscore, and their children, as
> raw text?
>
> Alternatively since there already is a `--latex-engine-opt` what about a
> `--filter-option=KEY:VAL`/`-O KEY=VAL`? Rather than passing its fields
> separately to the filters it can be included in the metadata under a key
> `filter-options`. It would differ from the rest of the metadata by not being
> specially interpreted, just a mapping of keys to plain strings. It would of
> course be desirable that fields (and possibly hierarchical children) of that
> mapping can be set in a metadata block as well, with fields given on the
> command line overriding those set in metadata blocks on a per-field basis.
> Would that be reasonable?
>
> FWIW I have written Perl code which I use in my filters which scans the
> metadata for keys which start with the filter name and walks the values of
> such fields extracting the first 'stringy' scalar value from each
> MetaInlines or MetaBlocks, returning a hash which mirrors the MetaMap and
> MetaList objects under those fields as plain hashes/arrays. The problem is
> that that code is often larger than the actual filter. I have turned it into
> a module, though.
>
> /bpj
>
>
> Den 2015-11-29 kl. 07:30, skrev John MacFarlane:
>>
>> +++ John MacFarlane [Nov 27 15 11:51 ]:
>>>
>>> I suppose the reason filter arguments weren't allowed is
>>> that pandoc treats the whole string (argument of --filter)
>>> as the command.  Since pandoc just gets one string as
>>> argument of --filter, it would need to do its own parsing
>>> of this string into a command, options and arguments in
>>> order for --filter='myfilter --arg1 --arg2' to work.
>>> That's the main complication that I see, but it's
>>> non-trivial.  Note that file paths can contain spaces and
>>> hyphen characters.  I suppose we could split the string
>>> at the first occurrence of '--', though, and let the first
>>> part be the filter, the rest (split on spaces) the
>>> arguments.
>>
>>
>> One more relevant thing.  Filters are passed the name of
>> the output format as the first argument.  So, if we allowed
>> options and arguments to be specified, we'd need to
>> rearrange things.  E.g.,
>>
>>     pandoc -t latex --filter 'myfilter --option1 --option optarg'
>>
>> would call the filter using
>>
>>     myfilter latex --option1 --option optarg
>>
>> This could be done but it's a bit ugly.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/565B6279.6060608%40gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

Re: support arguments on filters

[John MACFARLANE]

+++ Matthew Pickering [Nov 29 15 20:52 ]:
>I think any discussion about metadata is a red herring. I don't want
>to include this information in the document I am trying to process, I
>want to put it in a makefile. Adding it as metadata is indirect and
>annoying to have to deal with.

Of course metadata can be passed in via the command line
--metadata option (and doing it this way passes a literal string).

>Perhaps it would be possible to use "shell" rather than "proc" in this section?
>
>https://github.com/jgm/pandoc/blob/64b32e1e8194644df37a115200f929a9553c6d16/src/Text/Pandoc/Process.hs#L63

Possibly, but we do need to pass the output format name as
the first argument, so you'd still need to find the end of
the command name and insert this there.

Re: support arguments on filters

[Russ]

[-- Attachment #1.1: Type: text/plain, Size: 1501 bytes --]



On Sunday, 29 November 2015 20:52:49 UTC, Matthew Pickering wrote:
>
> I think any discussion about metadata is a red herring. I don't want 
> to include this information in the document I am trying to process, I 
> want to put it in a makefile. Adding it as metadata is indirect and 
> annoying to have to deal with. 
>
>
We do actually do just that, each set of documents (ebooks in our case) has 
an 'intro' file (Introduction or whatever), which has the metadata. This 
contains all the information to process the ebook in our makefile. The 
authors have no real problem with this as it is a simple set of metatadata 
at the start of their first document. Seems to work quite well. It also 
means you can then process for different outputs, we might produce an ePub 
or Mobi, but we might also write out a folder for upload to a markdown 
based website containing the full ebook test, intro and downloads.

Does this help?

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/4028a589-8698-482a-bcfc-488f0cde1bd1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 2086 bytes --]

Re: support arguments on filters

[BP Jonsson]

[-- Attachment #1: Type: text/plain, Size: 2745 bytes --]

So do I. I like to have everything in one place. Splitting the information
between different files only invites headache. I've even experimented with
putting command line options in a metadata mapping and letting a wrapper
extract and decode the YAML to get at them. Moreover some of my filters use
hierarchical configuration data, and then metadata is the only way.

/bpj
Den 30 nov 2015 10:51 skrev "Russ" <info-ckBjHRpK5r9MPTj2movljOTW4wlIGRCZ@public.gmane.org>:

>
>
> On Sunday, 29 November 2015 20:52:49 UTC, Matthew Pickering wrote:
>>
>> I think any discussion about metadata is a red herring. I don't want
>> to include this information in the document I am trying to process, I
>> want to put it in a makefile. Adding it as metadata is indirect and
>> annoying to have to deal with.
>>
>>
> We do actually do just that, each set of documents (ebooks in our case)
> has an 'intro' file (Introduction or whatever), which has the metadata.
> This contains all the information to process the ebook in our makefile. The
> authors have no real problem with this as it is a simple set of metatadata
> at the start of their first document. Seems to work quite well. It also
> means you can then process for different outputs, we might produce an ePub
> or Mobi, but we might also write out a folder for upload to a markdown
> based website containing the full ebook test, intro and downloads.
>
> Does this help?
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/4028a589-8698-482a-bcfc-488f0cde1bd1%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/4028a589-8698-482a-bcfc-488f0cde1bd1%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAFC_yuSzRFMk5%3D7eF%3DtRPCo5ZQe%2B5VLHmQiyueaoYCfojHvtGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #2: Type: text/html, Size: 3904 bytes --]

Re: support arguments on filters

[msprevak]

[-- Attachment #1.1: Type: text/plain, Size: 781 bytes --]

Just to add that panzer (https://github.com/msprev/panzer) allows pandoc 
filters to take multiple arguments, amongst other things.

The arguments, as well as the filters to be run, are specified in metadata.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ec92f0bf-7656-4033-b3d1-7b320d4d58e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 1216 bytes --]

Re: support arguments on filters

[Kolen Cheung]

[-- Attachment #1.1: Type: text/plain, Size: 926 bytes --]



I’m sorry to everyone in this thread: I absent-mindedly opened another 
thread on the same matter in 
https://groups.google.com/d/msg/pandoc-discuss/LIAfgkZKUiE/HWr_1k13EgAJ. At 
the time I realized the duplication, it was already too late. If you’re 
interested in the further discussion, please go to the link mentioned above.
​

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/bc314651-6555-422c-b5cf-7025a78f6ec9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 2284 bytes --]