Binary version of native AST instead of JSON

Binary version of native AST instead of JSON

[Ivan Kochurkin]

[-- Attachment #1.1: Type: text/plain, Size: 604 bytes --]

Maybe it makes sense to use Protobuf 
<https://developers.google.com/protocol-buffers>or MessagePack 
<https://msgpack.org/>format.

It can be important in cases where performance is critical.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1eb0d4b6-0500-4cf2-a3d0-d5442bb31afan%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 937 bytes --]

Re: Binary version of native AST instead of JSON

[John MacFarlane]

Ivan Kochurkin <ivan.kochurkin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> Maybe it makes sense to use Protobuf 
> <https://developers.google.com/protocol-buffers>or MessagePack 
> <https://msgpack.org/>format.
>
> It can be important in cases where performance is critical.

We could do this, but my guess is that it's not worth the
effort. The time it takes to serialize/deserialize JSON is
generally dwarfed by the other parts of the pandoc pipeline
(especially parsing).

Note that if performance is an issue, lua filters (which
avoid the JSON intermediate) will generally be quite a bit
faster than regular filters.

Re: Binary version of native AST instead of JSON

[Leonard Rosenthol]

[-- Attachment #1: Type: text/plain, Size: 2009 bytes --]

A binary serialization of the AST would only really be useful in the
context of storing it away for some reason - and you'd be better served
saving something like (compressed) pandoc markdown instead.

Also, be aware that protobuf and messagepack are both proprietary
specifications.  If you are looking for an open standard in this area, try
CBOR (https://tools.ietf.org/html/rfc7049).

Leonard


On Tue, Nov 24, 2020 at 1:07 PM John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:

>
> Ivan Kochurkin <ivan.kochurkin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
> > Maybe it makes sense to use Protobuf
> > <https://developers.google.com/protocol-buffers>or MessagePack
> > <https://msgpack.org/>format.
> >
> > It can be important in cases where performance is critical.
>
> We could do this, but my guess is that it's not worth the
> effort. The time it takes to serialize/deserialize JSON is
> generally dwarfed by the other parts of the pandoc pipeline
> (especially parsing).
>
> Note that if performance is an issue, lua filters (which
> avoid the JSON intermediate) will generally be quite a bit
> faster than regular filters.
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/m25z5ubpcq.fsf%40MacBook-Pro.hsd1.ca.comcast.net
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CALu%3Dv3LKt0aNX%3D8_SUWPsHorv1jJv2B1z9w2_qWgOHuULJA%3Dow%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 3337 bytes --]

Re: Binary version of native AST instead of JSON

[Ivan Kochurkin]

[-- Attachment #1.1: Type: text/plain, Size: 2392 bytes --]

Where did you find information about proprietary? Wikipedia says it's 
licensed under Apache <https://en.wikipedia.org/wiki/MessagePack>, Protobuf 
is licensed under BSD <https://en.wikipedia.org/wiki/Protocol_Buffers>.

вторник, 24 ноября 2020 г. в 22:24:27 UTC+3, Leonard Rosenthol: 

> A binary serialization of the AST would only really be useful in the 
> context of storing it away for some reason - and you'd be better served 
> saving something like (compressed) pandoc markdown instead.
>
> Also, be aware that protobuf and messagepack are both proprietary 
> specifications.  If you are looking for an open standard in this area, try 
> CBOR (https://tools.ietf.org/html/rfc7049).
>
> Leonard
>
>
> On Tue, Nov 24, 2020 at 1:07 PM John MacFarlane <j...-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
>
>>
>> Ivan Kochurkin <ivan.ko...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>
>> > Maybe it makes sense to use Protobuf 
>> > <https://developers.google.com/protocol-buffers>or MessagePack 
>> > <https://msgpack.org/>format.
>> >
>> > It can be important in cases where performance is critical.
>>
>> We could do this, but my guess is that it's not worth the
>> effort. The time it takes to serialize/deserialize JSON is
>> generally dwarfed by the other parts of the pandoc pipeline
>> (especially parsing).
>>
>> Note that if performance is an issue, lua filters (which
>> avoid the JSON intermediate) will generally be quite a bit
>> faster than regular filters.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/m25z5ubpcq.fsf%40MacBook-Pro.hsd1.ca.comcast.net
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/640f505c-39a8-4d2e-92e8-6d485b2cf41en%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 4700 bytes --]

Re: Binary version of native AST instead of JSON

[Leonard Rosenthol]

[-- Attachment #1: Type: text/plain, Size: 3369 bytes --]

You are referring to specific *implementations* of the specification.    I
am referring to the *specification* itself - which is controlled by Google
(Protobuf) and by an unnamed entity (MessagePack).

Leonard


On Tue, Nov 24, 2020 at 2:52 PM Ivan Kochurkin <ivan.kochurkin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:

> Where did you find information about proprietary? Wikipedia says it's
> licensed under Apache <https://en.wikipedia.org/wiki/MessagePack>,
> Protobuf is licensed under BSD
> <https://en.wikipedia.org/wiki/Protocol_Buffers>.
>
> вторник, 24 ноября 2020 г. в 22:24:27 UTC+3, Leonard Rosenthol:
>
>> A binary serialization of the AST would only really be useful in the
>> context of storing it away for some reason - and you'd be better served
>> saving something like (compressed) pandoc markdown instead.
>>
>> Also, be aware that protobuf and messagepack are both proprietary
>> specifications.  If you are looking for an open standard in this area, try
>> CBOR (https://tools.ietf.org/html/rfc7049).
>>
>> Leonard
>>
>>
>> On Tue, Nov 24, 2020 at 1:07 PM John MacFarlane <j...-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org>
>> wrote:
>>
>>>
>>> Ivan Kochurkin <ivan.ko...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>>
>>> > Maybe it makes sense to use Protobuf
>>> > <https://developers.google.com/protocol-buffers>or MessagePack
>>> > <https://msgpack.org/>format.
>>> >
>>> > It can be important in cases where performance is critical.
>>>
>>> We could do this, but my guess is that it's not worth the
>>> effort. The time it takes to serialize/deserialize JSON is
>>> generally dwarfed by the other parts of the pandoc pipeline
>>> (especially parsing).
>>>
>>> Note that if performance is an issue, lua filters (which
>>> avoid the JSON intermediate) will generally be quite a bit
>>> faster than regular filters.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "pandoc-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/pandoc-discuss/m25z5ubpcq.fsf%40MacBook-Pro.hsd1.ca.comcast.net
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/640f505c-39a8-4d2e-92e8-6d485b2cf41en%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/640f505c-39a8-4d2e-92e8-6d485b2cf41en%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CALu%3Dv3LGvMGgZAqLOdM%3DXDo39fL4NQADixtH7mhkZQvsbg3h%3Dw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 5389 bytes --]