Tools to format/visualize (JSON) AST

Tools to format/visualize (JSON) AST

[Stephan Boltzmann]

[-- Attachment #1.1: Type: text/plain, Size: 941 bytes --]

Hello everybody out there using pandoc,

A typical workflow for writing custom filters probably includes looking at 
the AST with the option "-t json", which is one long line by default.
I tried to command line tool "jq" to format the AST in a nicer way for easy 
navigation, but didn't find it too convenient in the end: The whole AST 
gets very long and I ended up using "grep" to look for the parts of 
interest.
I would be interested in which tools you are using for 
formatting/navigating the JSON AST.

Kind regards
Stephan

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/264bb687-2c7c-4c78-b173-b4e8eaca5728n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1995 bytes --]

Re: Tools to format/visualize (JSON) AST

[Zev Spitz]

[-- Attachment #1.1: Type: text/plain, Size: 1203 bytes --]

Not sure it helps, but I relied very heavily on Visual Studio's built-in 
JSON viewer while debugging PandocFilters. 
<https://github.com/zspitz/PandocFilters>

On Saturday, January 9, 2021 at 3:04:20 PM UTC+2 Stephan Boltzmann wrote:

> Hello everybody out there using pandoc,
>
> A typical workflow for writing custom filters probably includes looking at 
> the AST with the option "-t json", which is one long line by default.
> I tried to command line tool "jq" to format the AST in a nicer way for 
> easy navigation, but didn't find it too convenient in the end: The whole 
> AST gets very long and I ended up using "grep" to look for the parts of 
> interest.
> I would be interested in which tools you are using for 
> formatting/navigating the JSON AST.
>
> Kind regards
> Stephan
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/42b485cc-0712-43f9-aaf2-efbbb9a9cc29n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2490 bytes --]

Re: Tools to format/visualize (JSON) AST

[Stephan Boltzmann]

[-- Attachment #1.1: Type: text/plain, Size: 1476 bytes --]

Thanks a lot for the hint.
Since I'm working on Linux, I will probably not be able to use it, but it's 
good to know there's a Windows option.

spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org schrieb am Samstag, 9. Januar 2021 um 17:50:27 UTC+1:

> Not sure it helps, but I relied very heavily on Visual Studio's built-in 
> JSON viewer while debugging PandocFilters. 
> <https://github.com/zspitz/PandocFilters>
>
> On Saturday, January 9, 2021 at 3:04:20 PM UTC+2 Stephan Boltzmann wrote:
>
>> Hello everybody out there using pandoc,
>>
>> A typical workflow for writing custom filters probably includes looking 
>> at the AST with the option "-t json", which is one long line by default.
>> I tried to command line tool "jq" to format the AST in a nicer way for 
>> easy navigation, but didn't find it too convenient in the end: The whole 
>> AST gets very long and I ended up using "grep" to look for the parts of 
>> interest.
>> I would be interested in which tools you are using for 
>> formatting/navigating the JSON AST.
>>
>> Kind regards
>> Stephan
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/35ff7997-876c-4bd2-9b76-61cdafa0f0ban%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3187 bytes --]

Re: Tools to format/visualize (JSON) AST

[Julien Dutant]

[-- Attachment #1.1: Type: text/plain, Size: 1913 bytes --]

I've found useful to set up Zerobrane with mobdebug, place breakpoints and 
look at the stack explorer in Zerobrane (unfolds JSON elements in a 
tree-like manner). Otherwise used an online viewer 
(http://jsonviewer.stack.hu/) and pprint.lua 
(https://github.com/jagt/pprint.lua). Following, I'm sure others have 
better solutions.

On Saturday, January 9, 2021 at 5:03:16 PM UTC Stephan Boltzmann wrote:

> Thanks a lot for the hint.
> Since I'm working on Linux, I will probably not be able to use it, but 
> it's good to know there's a Windows option.
>
> spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org schrieb am Samstag, 9. Januar 2021 um 17:50:27 UTC+1:
>
>> Not sure it helps, but I relied very heavily on Visual Studio's built-in 
>> JSON viewer while debugging PandocFilters. 
>> <https://github.com/zspitz/PandocFilters>
>>
>> On Saturday, January 9, 2021 at 3:04:20 PM UTC+2 Stephan Boltzmann wrote:
>>
>>> Hello everybody out there using pandoc,
>>>
>>> A typical workflow for writing custom filters probably includes looking 
>>> at the AST with the option "-t json", which is one long line by default.
>>> I tried to command line tool "jq" to format the AST in a nicer way for 
>>> easy navigation, but didn't find it too convenient in the end: The whole 
>>> AST gets very long and I ended up using "grep" to look for the parts of 
>>> interest.
>>> I would be interested in which tools you are using for 
>>> formatting/navigating the JSON AST.
>>>
>>> Kind regards
>>> Stephan
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/e40dc90c-db0b-4d3e-a75a-4847dcd0f253n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3853 bytes --]

Re: Tools to format/visualize (JSON) AST

[William Lupton]

[-- Attachment #1.1: Type: text/plain, Size: 3569 bytes --]

I wrote a lua 'pandoc-aware inspector' that I find quite useful. See the
attached demo.lua (with a .txt extension in case any mailers don't
like .lua). I'd be happy to contribute this somewhere sensible.

With this doc:

-------- demo.md --------
# Demo document

This is a paragraph.
--------

it gives the following:

--------
% pandoc --lua-filter demo.lua demo.md >/dev/null
{
  blocks: {
    [1] Header {
      attr: Attr {
        [1] "demo-document"
        [2] {}
        [3] {}
        attributes: {}
        classes: {}
        identifier: "demo-document"
      }
      content: {
        [1] Str text: "Demo"
        [2] Space {}
        [3] Str text: "document"
      }
      level: 1
    }
    [2] content: Para {
      [1] Str text: "This"
      [2] Space {}
      [3] Str text: "is"
      [4] Space {}
      [5] Str text: "a"
      [6] Space {}
      [7] Str text: "paragraph."
    }
  }
  meta: {}
}
--------

On Sun, 10 Jan 2021 at 14:17, Julien Dutant <julien.dutant-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> I've found useful to set up Zerobrane with mobdebug, place breakpoints and
> look at the stack explorer in Zerobrane (unfolds JSON elements in a
> tree-like manner). Otherwise used an online viewer (
> http://jsonviewer.stack.hu/) and pprint.lua (
> https://github.com/jagt/pprint.lua). Following, I'm sure others have
> better solutions.
>
> On Saturday, January 9, 2021 at 5:03:16 PM UTC Stephan Boltzmann wrote:
>
>> Thanks a lot for the hint.
>> Since I'm working on Linux, I will probably not be able to use it, but
>> it's good to know there's a Windows option.
>>
>> spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org schrieb am Samstag, 9. Januar 2021 um 17:50:27 UTC+1:
>>
>>> Not sure it helps, but I relied very heavily on Visual Studio's built-in
>>> JSON viewer while debugging PandocFilters.
>>> <https://github.com/zspitz/PandocFilters>
>>>
>>> On Saturday, January 9, 2021 at 3:04:20 PM UTC+2 Stephan Boltzmann wrote:
>>>
>>>> Hello everybody out there using pandoc,
>>>>
>>>> A typical workflow for writing custom filters probably includes looking
>>>> at the AST with the option "-t json", which is one long line by
>>>> default.
>>>> I tried to command line tool "jq" to format the AST in a nicer way for
>>>> easy navigation, but didn't find it too convenient in the end: The whole
>>>> AST gets very long and I ended up using "grep" to look for the parts
>>>> of interest.
>>>> I would be interested in which tools you are using for
>>>> formatting/navigating the JSON AST.
>>>>
>>>> Kind regards
>>>> Stephan
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/e40dc90c-db0b-4d3e-a75a-4847dcd0f253n%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/e40dc90c-db0b-4d3e-a75a-4847dcd0f253n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxgcYNqv4CQwVDMYLm7iVz_yF2HsS9O4n8V%3De_D0J8ky8A%40mail.gmail.com.

[-- Attachment #1.2: Type: text/html, Size: 6337 bytes --]

[-- Attachment #2: demo.lua.txt --]
[-- Type: text/plain, Size: 4449 bytes --]

-- illustrate use of pandoc-aware inspector

-- version of pairs() with sorted keys
local function spairs(tab, comp)
    local snext = function(t, key)
        -- first invocation
        if key == nil then
            -- create sorted list with entries pointing to table entries
            local sorted = {}
            local i = 1
            for k, v in pairs(t) do
                sorted[i] = {k=k, v=v}
                i = i + 1
            end
            local cmp = comp and function(a, b) return comp(a.k, b.k) end
                or function(a, b) return a.k < b.k end
            table.sort(sorted, cmp)

            -- create index mapping key to next key
            local nextindex = {}
            for i, item in ipairs(sorted) do
                nextindex[item.k] = i + 1
            end

            -- store them on the table
            t.__sorted = sorted
            t.__nextindex = nextindex
        end

        -- get index from key
        local index = key == nil and 1 or t.__nextindex[key]

        -- get the correspond item from the sorted list
        local item = t.__sorted[index]

        -- if defined, return the item
        if item ~= nil then
            return item.k, item.v

        else
            -- otherwise tidy up
            t.__sorted = nil
            t.__nextindex = nil
            return nil
        end
    end

    -- standard iterator return
    return snext, tab, nil
end

-- helper function to return a sensible typename
-- XXX is there a better way of doing this? using metatables?
local function getTypename(value)
    return type(value) ~= 'table' and type(value) or (value.t and value.tag)
        and value.t or type(value)
end

-- helper function for pandoc-aware inspector
-- note: don't use anything from the pandoc module (so can test standalone)
-- XXX should detect repetition/recursion
-- XXX would like maxlen logic to apply at all levels? but not trivial
local function inspect_(prefix, value, maxlen, level, add)
    local buffer = {}
    if prefix == nil then prefix = '' end
    if level == nil then level = 0 end
    if add == nil then add = function(item) table.insert(buffer, item) end end
    local indent = maxlen and '' or ('  '):rep(level)
    local typename = getTypename(value)
    local typ = ({boolean=1, number=1, string=1, table=1})[typename] and '' or
        typename
    if type(value) == 'table' then
        local numKeys, lastKey = 0, nil
        for key in spairs(value) do
            numKeys = numKeys + 1
            lastKey = key
        end
        -- XXX this wouldn't be needed with proper maxlen handling
        if numKeys == 0 then
            value = '{}'
        elseif numKeys == 1 and lastKey == 'content' then
            typ = string.format('content: %s', typ)
            value = value.content
        elseif numKeys == 1 and lastKey == 'text' and typename == 'Str' then
            typename = 'string'
            typ = 'Str text:'
            value = value.text
        end
    end
    local presep = #prefix > 0 and ' ' or ''
    local typsep = #typ > 0 and ' ' or ''
    if type(value) ~= 'table' then
        local fmt = typename == 'string' and '%q' or '%s'
        add(string.format('%s%s%s%s%s' .. fmt, indent, prefix, presep,
                          typ, typsep, value))
    else
        add(string.format('%s%s%s%s%s{', indent, prefix, presep, typ, typsep))
        for i, val in ipairs(value) do
            local pre = maxlen and i > 1 and ', ' or ''
            local text = inspect_(string.format('%s[%s]', pre, i), val, maxlen,
                                  level + 1, add)
        end
        local first = true
        for key, val in spairs(value) do
            if not tonumber(key) then
                local pre = maxlen and not first and ', ' or ''
                local text = inspect_(string.format('%s%s:', pre, key), val,
                                      maxlen, level + 1, add)
            end
            first = false
        end
        add(string.format('%s}', indent))
    end
    return table.concat(buffer, maxlen and '' or '\n')
end

-- pandoc-aware inspector
local function inspect(value, maxlen)
    if maxlen == nil then maxlen = 70 end
    local text = inspect_(nil, value, maxlen)
    if #text > maxlen then
        text = inspect_(nil, value, nil)
    end
    return text
end

-- dump the entire document to stderr
function Pandoc(doc)
    io.stderr:write(inspect(doc) .. '\n')
end

Re: Tools to format/visualize (JSON) AST

[Zev Spitz]

[-- Attachment #1.1: Type: text/plain, Size: 3828 bytes --]

https://github.com/antonmedv/fx looks very nice.

On Sunday, January 10, 2021 at 7:34:47 PM UTC+2 William Lupton wrote:

> I wrote a lua 'pandoc-aware inspector' that I find quite useful. See the 
> attached demo.lua (with a .txt extension in case any mailers don't 
> like .lua). I'd be happy to contribute this somewhere sensible.
>
> With this doc:
>
> -------- demo.md --------
> # Demo document
>
> This is a paragraph.
> --------
>
> it gives the following:
>
> --------
> % pandoc --lua-filter demo.lua demo.md >/dev/null
> {
>   blocks: {
>     [1] Header {
>       attr: Attr {
>         [1] "demo-document"
>         [2] {}
>         [3] {}
>         attributes: {}
>         classes: {}
>         identifier: "demo-document"
>       }
>       content: {
>         [1] Str text: "Demo"
>         [2] Space {}
>         [3] Str text: "document"
>       }
>       level: 1
>     }
>     [2] content: Para {
>       [1] Str text: "This"
>       [2] Space {}
>       [3] Str text: "is"
>       [4] Space {}
>       [5] Str text: "a"
>       [6] Space {}
>       [7] Str text: "paragraph."
>     }
>   }
>   meta: {}
> }
> --------
>
> On Sun, 10 Jan 2021 at 14:17, Julien Dutant <julien...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> I've found useful to set up Zerobrane with mobdebug, place breakpoints 
>> and look at the stack explorer in Zerobrane (unfolds JSON elements in a 
>> tree-like manner). Otherwise used an online viewer (
>> http://jsonviewer.stack.hu/) and pprint.lua (
>> https://github.com/jagt/pprint.lua). Following, I'm sure others have 
>> better solutions.
>>
>> On Saturday, January 9, 2021 at 5:03:16 PM UTC Stephan Boltzmann wrote:
>>
>>> Thanks a lot for the hint.
>>> Since I'm working on Linux, I will probably not be able to use it, but 
>>> it's good to know there's a Windows option.
>>>
>>> spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org schrieb am Samstag, 9. Januar 2021 um 17:50:27 UTC+1:
>>>
>>>> Not sure it helps, but I relied very heavily on Visual Studio's 
>>>> built-in JSON viewer while debugging PandocFilters. 
>>>> <https://github.com/zspitz/PandocFilters>
>>>>
>>>> On Saturday, January 9, 2021 at 3:04:20 PM UTC+2 Stephan Boltzmann 
>>>> wrote:
>>>>
>>>>> Hello everybody out there using pandoc,
>>>>>
>>>>> A typical workflow for writing custom filters probably includes 
>>>>> looking at the AST with the option "-t json", which is one long line 
>>>>> by default.
>>>>> I tried to command line tool "jq" to format the AST in a nicer way 
>>>>> for easy navigation, but didn't find it too convenient in the end: The 
>>>>> whole AST gets very long and I ended up using "grep" to look for the 
>>>>> parts of interest.
>>>>> I would be interested in which tools you are using for 
>>>>> formatting/navigating the JSON AST.
>>>>>
>>>>> Kind regards
>>>>> Stephan
>>>>>
>>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/e40dc90c-db0b-4d3e-a75a-4847dcd0f253n%40googlegroups.com 
>> <https://groups.google.com/d/msgid/pandoc-discuss/e40dc90c-db0b-4d3e-a75a-4847dcd0f253n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/c57c3e4d-2fe9-4c62-8150-6038320afd00n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 7698 bytes --]

Re: Tools to format/visualize (JSON) AST

[Stephan Boltzmann]

[-- Attachment #1.1: Type: text/plain, Size: 4007 bytes --]

Thanks a lot for sharing the code. I like the inspector a lot, it works 
without having to install any dependencies.

https://github.com/antonmedv/fx is also interesting, particularly for 
people working a lot with Java Script.

William Lupton schrieb am Sonntag, 10. Januar 2021 um 18:34:47 UTC+1:

> I wrote a lua 'pandoc-aware inspector' that I find quite useful. See the 
> attached demo.lua (with a .txt extension in case any mailers don't 
> like .lua). I'd be happy to contribute this somewhere sensible.
>
> With this doc:
>
> -------- demo.md --------
> # Demo document
>
> This is a paragraph.
> --------
>
> it gives the following:
>
> --------
> % pandoc --lua-filter demo.lua demo.md >/dev/null
> {
>   blocks: {
>     [1] Header {
>       attr: Attr {
>         [1] "demo-document"
>         [2] {}
>         [3] {}
>         attributes: {}
>         classes: {}
>         identifier: "demo-document"
>       }
>       content: {
>         [1] Str text: "Demo"
>         [2] Space {}
>         [3] Str text: "document"
>       }
>       level: 1
>     }
>     [2] content: Para {
>       [1] Str text: "This"
>       [2] Space {}
>       [3] Str text: "is"
>       [4] Space {}
>       [5] Str text: "a"
>       [6] Space {}
>       [7] Str text: "paragraph."
>     }
>   }
>   meta: {}
> }
> --------
>
> On Sun, 10 Jan 2021 at 14:17, Julien Dutant <julien...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> I've found useful to set up Zerobrane with mobdebug, place breakpoints 
>> and look at the stack explorer in Zerobrane (unfolds JSON elements in a 
>> tree-like manner). Otherwise used an online viewer (
>> http://jsonviewer.stack.hu/) and pprint.lua (
>> https://github.com/jagt/pprint.lua). Following, I'm sure others have 
>> better solutions.
>>
>> On Saturday, January 9, 2021 at 5:03:16 PM UTC Stephan Boltzmann wrote:
>>
>>> Thanks a lot for the hint.
>>> Since I'm working on Linux, I will probably not be able to use it, but 
>>> it's good to know there's a Windows option.
>>>
>>> spit...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org schrieb am Samstag, 9. Januar 2021 um 17:50:27 UTC+1:
>>>
>>>> Not sure it helps, but I relied very heavily on Visual Studio's 
>>>> built-in JSON viewer while debugging PandocFilters. 
>>>> <https://github.com/zspitz/PandocFilters>
>>>>
>>>> On Saturday, January 9, 2021 at 3:04:20 PM UTC+2 Stephan Boltzmann 
>>>> wrote:
>>>>
>>>>> Hello everybody out there using pandoc,
>>>>>
>>>>> A typical workflow for writing custom filters probably includes 
>>>>> looking at the AST with the option "-t json", which is one long line 
>>>>> by default.
>>>>> I tried to command line tool "jq" to format the AST in a nicer way 
>>>>> for easy navigation, but didn't find it too convenient in the end: The 
>>>>> whole AST gets very long and I ended up using "grep" to look for the 
>>>>> parts of interest.
>>>>> I would be interested in which tools you are using for 
>>>>> formatting/navigating the JSON AST.
>>>>>
>>>>> Kind regards
>>>>> Stephan
>>>>>
>>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/e40dc90c-db0b-4d3e-a75a-4847dcd0f253n%40googlegroups.com 
>> <https://groups.google.com/d/msgid/pandoc-discuss/e40dc90c-db0b-4d3e-a75a-4847dcd0f253n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/8619be1c-f7dd-480d-b60f-509be3477ecdn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 7969 bytes --]