error using new python filters

error using new python filters

[ben]

[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]

Perhaps you can give me a hint as to how to fix this?

I'm trying to use the new python filters, and have dowloaded the graphviz 
example as a starting point. I can run this command:

    pandoc -f markdown -t json test.markdown | python graphviz.py




and get a blank image created, and this output:

    [{"unMeta": {"date": {"MetaInlines": [{"Str": "here"}]}, "author": {
"MetaList": [{"MetaInlines": [{"Str": "hoes"}]}]}, "title": {"MetaInlines": 
[{"Str": "Heading"}]}}}, [{"Para": [{"Str": "\"asdas"}, {"Space": []}, {
"RawInline": ["tex", "\\textsc{II}"]}, {"Space": []}, {"Str": "asdas\""}]}, 
{"BlockQuote": [{"Para": [{"Str": "\"sadsa\""}]}]}, {"Para": [{"Image": [[{
"Str": "caption"}], [
"graphviz-images/facdb053223830440541aeac62d7d3fc1a43edc6.png", ""]]}]}]]





Which indicates to me things are sort-of-ok, However if I try 

    pandoc --filter graphviz.py -f markdown -t latex test.markdown



Then I get this error:

    pandoc: Error running filter graphviz.py
    fd:4: hClose: resource vanished (Broken pipe)



Any hints much appreciated - I'm quite looking forward to using the new 
filters.

Ben


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/c3a05b34-7f73-44e8-901f-71a88c258ff5%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[-- Attachment #2: Type: text/html, Size: 13520 bytes --]

Re: error using new python filters

[John MacFarlane]

I assume you're using the dev version of pandoc.

>    Which indicates to me things are sort-of-ok, However if I try
>        pandoc --filter graphviz.py -f markdown -t latex test.markdown
>    Then I get this error:
>        pandoc: Error running filter graphviz.py
>        fd:4: hClose: resource vanished (Broken pipe)

Try ./graphviz.py

Otherwise it assumes the filter is in your path, which I assume
it is not...  Also make sure graphviz.py is executable (chmod +x).

+++ ben [Sep 05 13 08:58 ]:
>    Perhaps you can give me a hint as to how to fix this?
>    I'm trying to use the new python filters, and have dowloaded the
>    graphviz example as a starting point. I can run this command:
>        pandoc -f markdown -t json test.markdown | python graphviz.py
>    and get a blank image created, and this output:
>        [{"unMeta": {"date": {"MetaInlines": [{"Str": "here"}]}, "author":
>    {"MetaList": [{"MetaInlines": [{"Str": "hoes"}]}]}, "title":
>    {"MetaInlines": [{"Str": "Heading"}]}}}, [{"Para": [{"Str": "\"asdas"},
>    {"Space": []}, {"RawInline": ["tex", "\\textsc{II}"]}, {"Space": []},
>    {"Str": "asdas\""}]}, {"BlockQuote": [{"Para": [{"Str":
>    "\"sadsa\""}]}]}, {"Para": [{"Image": [[{"Str": "caption"}],
>    ["graphviz-images/facdb053223830440541aeac62d7d3fc1a43edc6.png",
>    ""]]}]}]]
>    Which indicates to me things are sort-of-ok, However if I try
>        pandoc --filter graphviz.py -f markdown -t latex test.markdown
>    Then I get this error:
>        pandoc: Error running filter graphviz.py
>        fd:4: hClose: resource vanished (Broken pipe)
>    Any hints much appreciated - I'm quite looking forward to using the new
>    filters.
>    Ben
> 
>    --
>    You received this message because you are subscribed to the Google
>    Groups "pandoc-discuss" group.
>    To unsubscribe from this group and stop receiving emails from it, send
>    an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>    To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>    To view this discussion on the web visit
>    [1]https://groups.google.com/d/msgid/pandoc-discuss/c3a05b34-7f73-44e8-
>    901f-71a88c258ff5%40googlegroups.com.
>    For more options, visit [2]https://groups.google.com/groups/opt_out.
> 
> References
> 
>    1. https://groups.google.com/d/msgid/pandoc-discuss/c3a05b34-7f73-44e8-901f-71a88c258ff5%40googlegroups.com
>    2. https://groups.google.com/groups/opt_out

Re: error using new python filters

[ben]

[-- Attachment #1: Type: text/plain, Size: 53 bytes --]

Thanks... it was the path issue causing the problem.

Re: error using new python filters

[ben]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

Just as some feedback, having played with this I think the ability to write 
filters in python is really nice, and sets the bar much lower than learning 
Haskell — I can see myself using it quite a bit. A few questions:

1. Obviously filters *can* live anywhere, but I wonder if they should, by 
default, be installed in ~/.pandoc/Filters? Perhaps also divided by 
language?

2. The pandoc.py module you provide is very helpful, and should perhaps be 
available via pip/pypy? However it's unfortunate that the package namespace 
now clashes with pyandoc (see https://pypi.python.org/pypi/pyandoc/0.0.1)   
Even though pyandoc isn't widely used, perhaps the new module could be 
renamed pandoc-filters?

3. Is there a way of accessing the document yaml metadata within a filter? 
Ditto the command line options.


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/307d269c-a2d9-4511-8ff2-d7d62b09e047%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[-- Attachment #2: Type: text/html, Size: 1757 bytes --]

Re: error using new python filters

[John MacFarlane]

+++ ben [Sep 05 13 15:28 ]:
> Just as some feedback, having played with this I think the ability to write 
> filters in python is really nice, and sets the bar much lower than learning 
> Haskell — I can see myself using it quite a bit. A few questions:
> 
> 1. Obviously filters *can* live anywhere, but I wonder if they should, by 
> default, be installed in ~/.pandoc/Filters? Perhaps also divided by 
> language?

Nothing stops you from putting them there and adding the directory to
your path.  So the question is really just whether pandoc should itself
add this directory to the path.  I think it's safer to leave that to the
user, but I'm open to persuasion.

> 2. The pandoc.py module you provide is very helpful, and should perhaps be 
> available via pip/pypy? However it's unfortunate that the package namespace 
> now clashes with pyandoc (see https://pypi.python.org/pypi/pyandoc/0.0.1)   
> Even though pyandoc isn't widely used, perhaps the new module could be 
> renamed pandoc-filters?

That's probably a good idea.  I can change the name of the module easily
enough.  What would I need to do to make it available via pip/pypy?  I
haven't worked much with python.

> 3. Is there a way of accessing the document yaml metadata within a filter? 
> Ditto the command line options.

In principle it ought to be possible to get the metadata, but the
current python module doesn't make this easy.  Let me think about how
to make it possible.

No, you can't access the command line options, although you do have
access to the output format.

Thanks for the feedback!

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/20130906004906.GA7557%40dhcp-128-32-252-11.lips.berkeley.edu.
For more options, visit https://groups.google.com/groups/opt_out.

Re: error using new python filters

[John MacFarlane]

+++ John MacFarlane [Sep 05 13 17:49 ]:
 
> > 2. The pandoc.py module you provide is very helpful, and should perhaps be 
> > available via pip/pypy? However it's unfortunate that the package namespace 
> > now clashes with pyandoc (see https://pypi.python.org/pypi/pyandoc/0.0.1)   
> > Even though pyandoc isn't widely used, perhaps the new module could be 
> > renamed pandoc-filters?
> 
> That's probably a good idea.  I can change the name of the module easily
> enough.  What would I need to do to make it available via pip/pypy?  I
> haven't worked much with python.

Hm, pandoc-filters won't work -- "modules should have short,
all-lowercase name" according to PEP 0008. Hyphens aren't allowed,
and underscores are discouraged.

So, we could use pandocfilters, which is ugly and not short.

Is there a policy against duplicating an existing module name in pip?
Nobody would be using pyandoc in a pandoc filter, I assume, so I
can't see any real threat of a clash in practice.

Re: error using new python filters

[John MacFarlane]

+++ John MacFarlane [Sep 05 13 17:49 ]:
> > 3. Is there a way of accessing the document yaml metadata within a filter? 
> > Ditto the command line options.
> 
> In principle it ought to be possible to get the metadata, but the
> current python module doesn't make this easy.  Let me think about how
> to make it possible.

I just pushed a change that allows you to define your action like this:

def myaction(key, value, format, **meta)

and then access the metadata fields using the meta dictionary.

Note, however, that the value returned won't be a string; it will
be a python object representing a pandoc MetaValue.  So, for
example, if your document starts out

% hi

then

meta['title'] == {"MetaInlines":[{"Str":"hi"}]}

Why don't you play with this and see if it works.

John

Re: error using new python filters

[Daniel Staal]

--As of September 5, 2013 5:49:06 PM -0700, John MacFarlane is alleged to 
have said:

> +++ ben [Sep 05 13 15:28 ]:
>> Just as some feedback, having played with this I think the ability to
>> write  filters in python is really nice, and sets the bar much lower
>> than learning  Haskell — I can see myself using it quite a bit. A few
>> questions:
>>
>> 1. Obviously filters *can* live anywhere, but I wonder if they should,
>> by  default, be installed in ~/.pandoc/Filters? Perhaps also divided by
>> language?
>
> Nothing stops you from putting them there and adding the directory to
> your path.  So the question is really just whether pandoc should itself
> add this directory to the path.  I think it's safer to leave that to the
> user, but I'm open to persuasion.

--As for the rest, it is mine.

I haven't been playing with filters, but my take on default locations like 
that would be that I would expect the program to *automatically* use files 
in that folder - which isn't my impression of how pandoc filters work.  If 
it's not automatic and it requires you to type the name of the file anyway, 
I'd expect you to want to have type the path as well.

Daniel T. Staal

---------------------------------------------------------------
This email copyright the author.  Unless otherwise noted, you
are expressly allowed to retransmit, quote, or otherwise use
the contents for non-commercial purposes.  This copyright will
expire 5 years after the author's death, or in 30 years,
whichever is longer, unless such a period is in excess of
local copyright law.
---------------------------------------------------------------

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/17823D12D67EC3D2F09BE460%40%5B192.168.1.50%5D.
For more options, visit https://groups.google.com/groups/opt_out.

Re: error using new python filters

[Joseph Reagle]

On 09/05/2013 09:15 PM, John MacFarlane wrote:
> So, we could use pandocfilters, which is ugly and not short.

pandocpy?

Re: error using new python filters

[ben]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]


>
> Why don't you play with this and see if it works. 
>


This looks great ... will do. Just to say you might also want to revise the 
signature of the example filters to to accept  **meta.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/4178608c-d391-451c-871e-9b3fa1ff6b28%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[-- Attachment #2: Type: text/html, Size: 1201 bytes --]

Re: error using new python filters

[John MacFarlane]

+++ ben [Sep 06 13 03:54 ]:
>      Why don't you play with this and see if it works.
> 
>    This looks great ... will do. Just to say you might also want to revise
>    the signature of the example filters to to accept  **meta.

I was thinking I'd add an example that actually uses **meta, but leave
**meta off the others -- after all, the whole reason I used **meta
instead of meta was to allow it to be left off by those who don't need
it.

Re: error using new python filters

[Jesse Rosenthal]

Thanks for adding this.

On Fri, 06 Sep 2013, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I was thinking I'd add an example that actually uses **meta, but leave
> **meta off the others -- after all, the whole reason I used **meta
> instead of meta was to allow it to be left off by those who don't need
> it.

It seems like the opposite would be the way to go. If every function
takes **meta, then they will work fine if the document doesn't have any
metadata (it will process that as no keywords, or an empty dict), as
well as when they do. If they don't take it, it will work fine without
any metadata, and then fail when metadata is added. So there's no real
loss by having it in the function parameters, but a real loss of
flexibility by keeping it out.

At the very least, you'll save yourself a lot of questions from people
downloading the filters, and wondering why they work on some documents
and not on others.

Best,
Jesse

Re: error using new python filters

[ben]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]


>
>
> It seems like the opposite would be the way to go. If every function 
> takes **meta, then they will work fine if the document doesn't have any 
> metadata (it will process that as no keywords, or an empty dict), as 
> well as when they do. If they don't take it, it will work fine without 
> any metadata, and then fail when metadata is added. So there's no real 
> loss by having it in the function parameters, but a real loss of 
> flexibility by keeping it out. 
>
>
I think this is right... the user only has to add a title with the % syntax 
and all the filters will then fail. 

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/a2b560c6-b7ff-4479-ab1e-eb62184477cd%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[-- Attachment #2: Type: text/html, Size: 1602 bytes --]

Re: error using new python filters

[ben]

[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]


>
> Why don't you play with this and see if it works. 
>


I've been using **meta  within filters and achieving most of what I want 
very easily. A few things have cropped up in the process:

1. Is there a was to add a class to an arbitrary block of markdown without 
using <div> tags?  Fenced code blocks kind of allow this, but the contents 
are then verbatim if converting to latex.
2. Would it be possible for filters to do their work and then mark a block 
of the document as 'needs re-processing'— i.e., effectively re-call the 
markdown parser? This would avoid the slightly hacky need to set up if/case 
structures to do different things with filtered text depending on the 
output format. 
3. Is there a description of the internal representation anywhere? It's 
obviously regular and one can work out what is happening from some test 
documents, but it would be nice. 
4. Somewhat related to 3, would it be possible to include helper functions 
in the pandoc.py module to do some common transformations on the native 
documents? For example, I think it's going to be a common case to want to 
extract the value of json meta data fields as a plain text string to work 
with in filters. Similarly, when the filter is performing some 
transformation on a chunk of text it might be nice to have simple access to 
the plain text.


Anyway - thanks again for including the python filters. All very useful.

B

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/41f5f129-2de1-4534-a62e-c7be7d49a6c9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[-- Attachment #2: Type: text/html, Size: 2544 bytes --]

Re: error using new python filters

[John MacFarlane]

+++ ben [Sep 10 13 03:52 ]:
>      Why don't you play with this and see if it works.
> 
>    I've been using **meta  within filters and achieving most of what I
>    want very easily. A few things have cropped up in the process:
>    1. Is there a was to add a class to an arbitrary block of markdown
>    without using <div> tags?  Fenced code blocks kind of allow this, but
>    the contents are then verbatim if converting to latex.

No, `<div>` tags are currently the only way.  Look back on this list
and you'll find extensive discussion of other possibilities.  I don't
rule out another syntax in the future, but there wasn't much agreement.

>    2. Would it be possible for filters to do their work and then mark a
>    block of the document as 'needs re-processing' i.e., effectively
>    re-call the markdown parser? This would avoid the slightly hacky need
>    to set up if/case structures to do different things with filtered text
>    depending on the output format.

No, this seems overly complex to me, and some aspects of parsing
cannot be ported well to parts of the document (e.g. resolution of
reference links).  What is unsatisfactory about the current setup?
Some filters will be output-format agnostic --  e.g. changing an Emph
to a Strong -- and they don't need to deal with 'format' at all.
Others aren't, and they'll need to deal with 'format'.  Maybe you could
describe the case you have in mind.

>    3. Is there a description of the internal representation anywhere? It's
>    obviously regular and one can work out what is happening from some test
>    documents, but it would be nice.

No, there isn't.  You can work it out from a few examples plus the
Text.Pandoc.Definition documentation on Hackage.  Perhaps one could
write a Haskell program to automatically generate some documentation of
the JSON format.  Right now I'm too busy; I need to focus on getting
1.12 out the door.

>    4. Somewhat related to 3, would it be possible to include helper
>    functions in the pandoc.py module to do some common transformations on
>    the native documents? For example, I think it's going to be a common
>    case to want to extract the value of json meta data fields as a plain
>    text string to work with in filters. Similarly, when the filter is
>    performing some transformation on a chunk of text it might be nice to
>    have simple access to the plain text.

Yes, it could be good to have a 'tostring' function to extract the
string content of metavalues.  (Note that this wouldn't correspond
to the rendered content, which might include formatting appropriate
to the output format; it would just extract contents of Str, Space,
and Code elements and concatenate them.)

This shouldn't be too hard to add.  (And it could be added after
the 1.12 release, since it's not an official part of pandoc.)

John

Re: error using new python filters

[ben]

[-- Attachment #1: Type: text/plain, Size: 4366 bytes --]


>
> No, `<div>` tags are currently the only way.  Look back on this list 
> and you'll find extensive discussion of other possibilities.  I don't 
> rule out another syntax in the future, but there wasn't much agreement. 
>
>
I'll have a peek in the archives. I can see that making it 'markdownish' is 
probably hard though. Perhaps this discussion could be resurrected after 
1.12? I think markdown really does need a simple attributes system -- there 
are loads of cases like sizing pictures where the current solutions are 
ugly – but ultimately you will have to make a decision... it's unlikely to 
get in the way of standard usages in any case.

 

> >    2. Would it be possible for filters to do their work and then mark a 
> >    block of the document as 'needs re-processing' i.e., effectively 
> >    re-call the markdown parser? This would avoid the slightly hacky need 
> >    to set up if/case structures to do different things with filtered 
> text 
> >    depending on the output format. 
>
> No, this seems overly complex to me, and some aspects of parsing 
> cannot be ported well to parts of the document (e.g. resolution of 
> reference links).  What is unsatisfactory about the current setup? 
> Some filters will be output-format agnostic --  e.g. changing an Emph 
> to a Strong -- and they don't need to deal with 'format' at all. 
> Others aren't, and they'll need to deal with 'format'.  Maybe you could 
> describe the case you have in mind. 
>

Fair point – when I think about it, the main case would be to  either 
address the problem of adding classes to elements by abusing the fenced 
code blocks, so this isn't a good reason to do it.

 

> >    3. Is there a description of the internal representation anywhere? 
> It's 
> >    obviously regular and one can work out what is happening from some 
> test 
> >    documents, but it would be nice. 
>
> No, there isn't.  You can work it out from a few examples plus the 
> Text.Pandoc.Definition documentation on Hackage.  Perhaps one could 
> write a Haskell program to automatically generate some documentation of 
> the JSON format.  Right now I'm too busy; I need to focus on getting 
> 1.12 out the door. 
>

Fair enough!  I just saw your list of publications so am impressed pandoc 
happens at all.

 

>
> >    4. Somewhat related to 3, would it be possible to include helper 
> >    functions in the pandoc.py module to do some common transformations 
> on 
> >    the native documents? For example, I think it's going to be a common 
> >    case to want to extract the value of json meta data fields as a plain 
> >    text string to work with in filters. Similarly, when the filter is 
> >    performing some transformation on a chunk of text it might be nice to 
> >    have simple access to the plain text. 
>
> Yes, it could be good to have a 'tostring' function to extract the 
> string content of metavalues.  (Note that this wouldn't correspond 
> to the rendered content, which might include formatting appropriate 
> to the output format; it would just extract contents of Str, Space, 
> and Code elements and concatenate them.) 
>

There's probably a nicer way, but I've been using this to process items in 
the 'unMeta' element of the json format:

def get_meta_tuple(metaitem):
        k, content = metaitem
        if "MetaInlines" in content:
            isuseful = lambda x: bool([i in x for i in ['Str', 'Space', 
'Code']])
            fixspace = lambda x: 'Space' in x and " " or x.values()[0]
            return (k, "".join(map(fixspace, filter(isuseful, content[
'MetaInlines']))))


        if "MetaBool" in content:
            return (k, content['MetaBool'])



Cheers,

Ben 

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/7a191625-da7b-4ecd-8e0d-af21c30b4615%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[-- Attachment #2: Type: text/html, Size: 12490 bytes --]

Re: error using new python filters

[John MacFarlane]

+++ ben [Sep 11 13 03:10 ]:
>      No, `<div>` tags are currently the only way.  Look back on this list
>      and you'll find extensive discussion of other possibilities.  I
>      don't
>      rule out another syntax in the future, but there wasn't much
>      agreement.
> 
>    I'll have a peek in the archives. I can see that making it
>    'markdownish' is probably hard though. Perhaps this discussion could be
>    resurrected after 1.12? I think markdown really does need a simple
>    attributes system -- there are loads of cases like sizing pictures
>    where the current solutions are ugly but ultimately you will have to
>    make a decision... it's unlikely to get in the way of standard usages
>    in any case.

I'm definitely open to further discussion.  The huge advantage of
`<span>` and `<div>` is that they're backward compatible with standard
markdown.

If we did do something else, I'd want to reuse the attribute syntax
now used in headers and code blocks. (Ditto for sizing pictures, but
there you have the additional issue that different sizing units are
appropriate for different formats.)

 
>      >    4. Somewhat related to 3, would it be possible to include
>      helper
>      >    functions in the pandoc.py module to do some common
>      transformations on
>      >    the native documents? For example, I think it's going to be a
>      common
>      >    case to want to extract the value of json meta data fields as a
>      plain
>      >    text string to work with in filters. Similarly, when the filter
>      is
>      >    performing some transformation on a chunk of text it might be
>      nice to
>      >    have simple access to the plain text.
>      Yes, it could be good to have a 'tostring' function to extract the
>      string content of metavalues.  (Note that this wouldn't correspond
>      to the rendered content, which might include formatting appropriate
>      to the output format; it would just extract contents of Str, Space,
>      and Code elements and concatenate them.)
> 
>    There's probably a nicer way, but I've been using this to process items
>    in the 'unMeta' element of the json format:
>    def get_meta_tuple(metaitem):
>            k, content = metaitem
>            if "MetaInlines" in content:
>                isuseful = lambda x: bool([i in x for i in ['Str', 'Space',
>    'Code']])
>                fixspace = lambda x: 'Space' in x and " " or x.values()[0]
>                return (k, "".join(map(fixspace, filter(isuseful,
>    content['MetaInlines']))))
>            if "MetaBool" in content:
>                return (k, content['MetaBool'])

I've added a 'stringify' function that converts a list of inline or
block elements to a string.  (Unlike your solution, it will recurse
into nested constructs:  'hi *there **bold** ok*' should turn into
'hi there bold ok'.)

It may not be quite what you need, though, as I'm not sure it will
operate directly on a metadata value (rather, on its contents); also,
it will ignore boolean values.  So maybe it would be good to export
a function that returns a simple (string or boolean) value for a
metadata field.

Re: error using new python filters

[ben]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]


>
> I've added a 'stringify' function that converts a list of inline or 
> block elements to a string.  (Unlike your solution, it will recurse 
> into nested constructs:  'hi *there **bold** ok*' should turn into 
> 'hi there bold ok'.) 
>
> It may not be quite what you need, though, as I'm not sure it will 
> operate directly on a metadata value (rather, on its contents); also, 
> it will ignore boolean values.  
>


That's nice that it recurses, although in this case I was just using it for 
settings so anything other than a string would have been an error really.
 

So maybe it would be good to export 
> a function that returns a simple (string or boolean) value for a 
> metadata field. 


Yes this would be a good idea.

Ben 

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/c8aa7646-e3b2-41e4-8677-3cbe66479858%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[-- Attachment #2: Type: text/html, Size: 2583 bytes --]

Re: error using new python filters

[John MacFarlane]

+++ ben [Sep 12 13 02:59 ]:
>      I've added a 'stringify' function that converts a list of inline or
>      block elements to a string.  (Unlike your solution, it will recurse
>      into nested constructs:  'hi *there **bold** ok*' should turn into
>      'hi there bold ok'.)
>      It may not be quite what you need, though, as I'm not sure it will
>      operate directly on a metadata value (rather, on its contents);
>      also,
>      it will ignore boolean values.
> 
>    That's nice that it recurses, although in this case I was just using it
>    for settings so anything other than a string would have been an error
>    really.
> 
>      So maybe it would be good to export
>      a function that returns a simple (string or boolean) value for a
>      metadata field.
> 
>    Yes this would be a good idea.

Note:  I've changed the repository name to pandocfilters.

I've also put this on PyPI, so you should now be able to
pip install pandocfilters.

Re: error using new python filters

[Richard Careaga]

[-- Attachment #1.1: Type: text/plain, Size: 2551 bytes --]

For the benefit of future pilgrims using any flavor of pandoc filter, the 
broken pipe error message is often seen. The preferred way of filtering is 
to use the filter flag in pandoc 

pandoc-f[format]-t[format]--filter[PATH/filter[filename]

The filter executable receives a serialized abstract syntax tree like the 
one shown in the OP, which must be manipulated by walking the tree, making 
the required modifications and then given back to the pandoc API for 
further processing. This is easy to do with the ENTIRE source document 
(i.e., applying the filter to all elements in the AST), but it is a royal 
pita to do it for only select elements. (At least, I haven't figured out 
yet how to extend the few examples).

On Thursday, September 5, 2013 at 8:58:12 AM UTC-7, ben wrote:
>
> Perhaps you can give me a hint as to how to fix this?
>
> I'm trying to use the new python filters, and have dowloaded the graphviz 
> example as a starting point. I can run this command:
>
>     pandoc -f markdown -t json test.markdown | python graphviz.py
>
>
>
>
> and get a blank image created, and this output:
>
>     [{"unMeta": {"date": {"MetaInlines": [{"Str": "here"}]}, "author": {
> "MetaList": [{"MetaInlines": [{"Str": "hoes"}]}]}, "title": {"MetaInlines"
> : [{"Str": "Heading"}]}}}, [{"Para": [{"Str": "\"asdas"}, {"Space": []}, {
> "RawInline": ["tex", "\\textsc{II}"]}, {"Space": []}, {"Str": "asdas\""
> }]}, {"BlockQuote": [{"Para": [{"Str": "\"sadsa\""}]}]}, {"Para": [{
> "Image": [[{"Str": "caption"}], [
> "graphviz-images/facdb053223830440541aeac62d7d3fc1a43edc6.png", ""]]}]}]]
>
>
>
>
>
> Which indicates to me things are sort-of-ok, However if I try 
>
>     pandoc --filter graphviz.py -f markdown -t latex test.markdown
>
>
>
> Then I get this error:
>
>     pandoc: Error running filter graphviz.py
>     fd:4: hClose: resource vanished (Broken pipe)
>
>
>
> Any hints much appreciated - I'm quite looking forward to using the new 
> filters.
>
> Ben
>
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/5be3403f-dd71-46e8-a7e9-1261bae44f89%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 11071 bytes --]

fd:3: hClose: resource vanished (Broken pipe) (was: Re: error using new python filters)

[BP Jonsson]

For reference: this can happen if there are two executables with 
the same name in PATH

/bpj

Den 2013-09-05 kl. 18:26, skrev John MacFarlane:
> I assume you're using the dev version of pandoc.
>
>>     Which indicates to me things are sort-of-ok, However if I try
>>         pandoc --filter graphviz.py -f markdown -t latex test.markdown
>>     Then I get this error:
>>         pandoc: Error running filter graphviz.py
>>         fd:4: hClose: resource vanished (Broken pipe)
>
> Try ./graphviz.py
>
> Otherwise it assumes the filter is in your path, which I assume
> it is not...  Also make sure graphviz.py is executable (chmod +x).
>
> +++ ben [Sep 05 13 08:58 ]:
>>     Perhaps you can give me a hint as to how to fix this?
>>     I'm trying to use the new python filters, and have dowloaded the
>>     graphviz example as a starting point. I can run this command:
>>         pandoc -f markdown -t json test.markdown | python graphviz.py
>>     and get a blank image created, and this output:
>>         [{"unMeta": {"date": {"MetaInlines": [{"Str": "here"}]}, "author":
>>     {"MetaList": [{"MetaInlines": [{"Str": "hoes"}]}]}, "title":
>>     {"MetaInlines": [{"Str": "Heading"}]}}}, [{"Para": [{"Str": "\"asdas"},
>>     {"Space": []}, {"RawInline": ["tex", "\\textsc{II}"]}, {"Space": []},
>>     {"Str": "asdas\""}]}, {"BlockQuote": [{"Para": [{"Str":
>>     "\"sadsa\""}]}]}, {"Para": [{"Image": [[{"Str": "caption"}],
>>     ["graphviz-images/facdb053223830440541aeac62d7d3fc1a43edc6.png",
>>     ""]]}]}]]
>>     Which indicates to me things are sort-of-ok, However if I try
>>         pandoc --filter graphviz.py -f markdown -t latex test.markdown
>>     Then I get this error:
>>         pandoc: Error running filter graphviz.py
>>         fd:4: hClose: resource vanished (Broken pipe)
>>     Any hints much appreciated - I'm quite looking forward to using the new
>>     filters.
>>     Ben
>>
>>     --
>>     You received this message because you are subscribed to the Google
>>     Groups "pandoc-discuss" group.
>>     To unsubscribe from this group and stop receiving emails from it, send
>>     an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>     To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>     To view this discussion on the web visit
>>     [1]https://groups.google.com/d/msgid/pandoc-discuss/c3a05b34-7f73-44e8-
>>     901f-71a88c258ff5%40googlegroups.com.
>>     For more options, visit [2]https://groups.google.com/groups/opt_out.
>>
>> References
>>
>>     1. https://groups.google.com/d/msgid/pandoc-discuss/c3a05b34-7f73-44e8-901f-71a88c258ff5%40googlegroups.com
>>     2. https://groups.google.com/groups/opt_out
>