Sample documents for testing JSON filters

Sample documents for testing JSON filters

[Zev Spitz]

[-- Attachment #1.1: Type: text/plain, Size: 675 bytes --]

I am trying to develop a .NET package that can be used for writing 
JSON-based input filters. Can anyone recommend a sample document that I can 
use to see that the JSON is being parsed correctly?
Alternatively, is there a set of tests which filters have to pass?

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/5e4c5ac9-b24c-4d4e-a0b3-3a094c263555n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 983 bytes --]

Re: Sample documents for testing JSON filters

[John MacFarlane]

I guess the most basic test is to make sure that your
deserialize composed with your serialize gives the
same document again.

Zev Spitz <spitzzev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> I am trying to develop a .NET package that can be used for writing 
> JSON-based input filters. Can anyone recommend a sample document that I can 
> use to see that the JSON is being parsed correctly?
> Alternatively, is there a set of tests which filters have to pass?
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/5e4c5ac9-b24c-4d4e-a0b3-3a094c263555n%40googlegroups.com.

Re: Sample documents for testing JSON filters

[christi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]

[-- Attachment #1.1: Type: text/plain, Size: 1380 bytes --]

In pandoc repo, under test dir, there’sa lot of native files. You can then 
use pandoc to convert them to JSON as a starting point.

Also, you could consider using other filter framework as a starting point. 
Not as something you use in the long term, but as you first develop your 
framework as part of validation phase.

One example would be panflute in Python. You can feed in JSON to it using 
convert_text. And if it fail to parse it, it will emits useful debug 
information, Eg telling you a certain element is expecting a Block but get 
an Inline instead.

On Sunday, November 22, 2020 at 1:48:26 PM UTC-8 spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:

> I am trying to develop a .NET package that can be used for writing 
> JSON-based input filters. Can anyone recommend a sample document that I can 
> use to see that the JSON is being parsed correctly?
> Alternatively, is there a set of tests which filters have to pass?
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/f2c7aa8e-6bfe-4b9a-89e8-adbdcd1225a8n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1883 bytes --]

Re: Sample documents for testing JSON filters

[Zev Spitz]

[-- Attachment #1.1: Type: text/plain, Size: 2011 bytes --]

I think I will use the files in the test directory. I'll pass them through 
pandoc first without a filter, then with a filter that does nothing, to 
ensure the AST is the same.
I'm assuming files that end with .native are the produced AST 
representations for each file, and the following folders don't have native 
files:

* Tests - contains the Haskell test runner
* lua
* ipynb
* media

1. What are the writer.* files?
2. What's in docx/golden?

Thanks in advance.

On Tuesday, November 24, 2020 at 4:07:27 AM UTC+2 christi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org 
wrote:

> In pandoc repo, under test dir, there’sa lot of native files. You can then 
> use pandoc to convert them to JSON as a starting point.
>
> Also, you could consider using other filter framework as a starting point. 
> Not as something you use in the long term, but as you first develop your 
> framework as part of validation phase.
>
> One example would be panflute in Python. You can feed in JSON to it using 
> convert_text. And if it fail to parse it, it will emits useful debug 
> information, Eg telling you a certain element is expecting a Block but get 
> an Inline instead.
>
> On Sunday, November 22, 2020 at 1:48:26 PM UTC-8 spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
>
>> I am trying to develop a .NET package that can be used for writing 
>> JSON-based input filters. Can anyone recommend a sample document that I can 
>> use to see that the JSON is being parsed correctly?
>> Alternatively, is there a set of tests which filters have to pass?
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/b16cdc3d-2be8-4ca5-92c4-137a53834b26n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2877 bytes --]

Re: Sample documents for testing JSON filters

[Kolen Cheung]

[-- Attachment #1: Type: text/plain, Size: 3275 bytes --]

Anything pandoc can convert to and from can be a test file. Native is just one of them, Eg you can generate JSON by `pandoc ...ipynb -t json` And it will use the ipynb reader.

Lua is a special case probably for testing lua filtering: `pandoc somedoc -F...lua`, probably not useful to you.

One thing to beware of is the license. Pandoc is GPLv2. If you decided to include the test files in your distributed code then it must be GPLv2. On GitHub the automatic archive in Releases include them automatically. But if you’re careful that the binary and source you distribute excludes those test file it will be ok. (IANAL)
On Dec 12, 2020, 12:26 PM -0800, Zev Spitz <spitzzev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, wrote:
> I think I will use the files in the test directory. I'll pass them through pandoc first without a filter, then with a filter that does nothing, to ensure the AST is the same.
> I'm assuming files that end with .native are the produced AST representations for each file, and the following folders don't have native files:
>
> * Tests - contains the Haskell test runner
> * lua
> * ipynb
> * media
>
> 1. What are the writer.* files?
> 2. What's in docx/golden?
>
> Thanks in advance.
>
> > On Tuesday, November 24, 2020 at 4:07:27 AM UTC+2 christi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> > > In pandoc repo, under test dir, there’sa lot of native files. You can then use pandoc to convert them to JSON as a starting point.
> > >
> > > Also, you could consider using other filter framework as a starting point. Not as something you use in the long term, but as you first develop your framework as part of validation phase.
> > >
> > > One example would be panflute in Python. You can feed in JSON to it using convert_text. And if it fail to parse it, it will emits useful debug information, Eg telling you a certain element is expecting a Block but get an Inline instead.
> > >
> > > > On Sunday, November 22, 2020 at 1:48:26 PM UTC-8 spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
> > > > > I am trying to develop a .NET package that can be used for writing JSON-based input filters. Can anyone recommend a sample document that I can use to see that the JSON is being parsed correctly?
> > > > > Alternatively, is there a set of tests which filters have to pass?
> --
> You received this message because you are subscribed to a topic in the Google Groups "pandoc-discuss" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/pandoc-discuss/gQRR9qat9Zw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/b16cdc3d-2be8-4ca5-92c4-137a53834b26n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d6898de4-430d-4bf9-9733-273f03220ede%40Spark.

[-- Attachment #2: Type: text/html, Size: 4931 bytes --]

Re: Sample documents for testing JSON filters

[Zev Spitz]

[-- Attachment #1.1: Type: text/plain, Size: 4312 bytes --]

I've implemented tests against the test documents in the Pandoc repo, 
testing that the JSON is equivalent before and after the filter, and 
testing that the AST is the same before and after the filter; the JSON test 
can be see here 
<https://github.com/zspitz/PandocFilters/blob/main/Tests/TestContainer.cs#L175>, 
while the AST test can be seen here 
<https://github.com/zspitz/PandocFilters/blob/main/Tests/TestContainer.cs#L150>
.
I'm still getting warnings 
<https://github.com/zspitz/PandocFilters/issues/11> for some files when 
trying to initially pass them into Pandoc; hopefully I'll be able to 
resolve them in time.
Thanks for the suggestions.
On Sunday, December 13, 2020 at 12:59:14 AM UTC+2 christi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org 
wrote:

> Anything pandoc can convert to and from can be a test file. Native is just 
> one of them, Eg you can generate JSON by `pandoc ...ipynb -t json` And it 
> will use the ipynb reader.
>
> Lua is a special case probably for testing lua filtering: `pandoc somedoc 
> -F...lua`, probably not useful to you.
>
> One thing to beware of is the license. Pandoc is GPLv2. If you decided to 
> include the test files in your distributed code then it must be GPLv2. On 
> GitHub the automatic archive in Releases include them automatically. But if 
> you’re careful that the binary and source you distribute excludes those 
> test file it will be ok. (IANAL)
> On Dec 12, 2020, 12:26 PM -0800, Zev Spitz <spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, wrote:
>
> I think I will use the files in the test directory. I'll pass them through 
> pandoc first without a filter, then with a filter that does nothing, to 
> ensure the AST is the same.
> I'm assuming files that end with .native are the produced AST 
> representations for each file, and the following folders don't have native 
> files:
>
> * Tests - contains the Haskell test runner
> * lua
> * ipynb
> * media
>
> 1. What are the writer.* files?
> 2. What's in docx/golden?
>
> Thanks in advance.
>
> On Tuesday, November 24, 2020 at 4:07:27 AM UTC+2 christi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org 
> wrote:
>
>> In pandoc repo, under test dir, there’sa lot of native files. You can 
>> then use pandoc to convert them to JSON as a starting point. 
>>
>> Also, you could consider using other filter framework as a starting 
>> point. Not as something you use in the long term, but as you first develop 
>> your framework as part of validation phase.
>>
>> One example would be panflute in Python. You can feed in JSON to it using 
>> convert_text. And if it fail to parse it, it will emits useful debug 
>> information, Eg telling you a certain element is expecting a Block but get 
>> an Inline instead.
>>
>> On Sunday, November 22, 2020 at 1:48:26 PM UTC-8 spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:
>>
>>> I am trying to develop a .NET package that can be used for writing 
>>> JSON-based input filters. Can anyone recommend a sample document that I can 
>>> use to see that the JSON is being parsed correctly?
>>> Alternatively, is there a set of tests which filters have to pass?
>>>
>> --
> You received this message because you are subscribed to a topic in the 
> Google Groups "pandoc-discuss" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/pandoc-discuss/gQRR9qat9Zw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pandoc-discuss/b16cdc3d-2be8-4ca5-92c4-137a53834b26n%40googlegroups.com 
> <https://groups.google.com/d/msgid/pandoc-discuss/b16cdc3d-2be8-4ca5-92c4-137a53834b26n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/f4425367-60d0-49be-ba4f-b0984b4d981dn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 6757 bytes --]