Feature Idea: docx -> HTML table styling

Feature Idea: docx -> HTML table styling

[Noah Malmed]

[-- Attachment #1.1: Type: text/plain, Size: 1713 bytes --]



Hello!

 We use Pandoc often to convert from docx to HTML, and many of the 
documents we convert include tables. As far as we can tell, almost all of 
the table styling is lost in the docx reader. Specifically, we care about 5 
things:

1. Text justification (left, center, or right)

2. Vertical alignment (top, middle, or bottom)

3. Text indentation

4. Cell shading and text color

5. Table borders 

We hope to enhance the docx reader so that these stylings get preserved in 
the AST.

Proposed solutions:

1. It seems like text justification  already exists in the AST through the 
Alignment value. It just needs to get implemented in the docx reader, as 
described in this issue: https://github.com/jgm/pandoc/issues/6316

2. Add the vertical alignment style to attributes as suggested here 
<https://github.com/jgm/pandoc/issues/7444#issuecomment-881649125>

3. Add text indentation to attributes in the form of the style padding-left

4. Add cell shading and text color to attributes in the form of the styles 
background-color and color

5. Add table borders to attributes in the form of the style border

 

Does this sound like a sane and feasible solution? We're pretty motivated 
and willing to work on these changes, just want to know if they would be 
the best route!

Thanks! 

Noah

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/cf7005a8-0447-4667-acb2-c1eccbaacaden%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2394 bytes --]

Re: Feature Idea: docx -> HTML table styling

[John MacFarlane]

Noah Malmed <nmalmed-O2gogPphfo5dNrB6XyqITwC/G2K4zDHf@public.gmane.org> writes:

> Hello!
>
>  We use Pandoc often to convert from docx to HTML, and many of the 
> documents we convert include tables. As far as we can tell, almost all of 
> the table styling is lost in the docx reader. Specifically, we care about 5 
> things:
>
> 1. Text justification (left, center, or right)
>
> 2. Vertical alignment (top, middle, or bottom)
>
> 3. Text indentation
>
> 4. Cell shading and text color
>
> 5. Table borders 
>
> We hope to enhance the docx reader so that these stylings get preserved in 
> the AST.
>
> Proposed solutions:
>
> 1. It seems like text justification  already exists in the AST through the 
> Alignment value. It just needs to get implemented in the docx reader, as 
> described in this issue: https://github.com/jgm/pandoc/issues/6316

Correct.

> 2. Add the vertical alignment style to attributes as suggested here 
> <https://github.com/jgm/pandoc/issues/7444#issuecomment-881649125>

Should be okay.  However, adding `vertical-align` there won't do
any good for converting to HTML unless the HTML writer is
modified to be sensitive to this attribute.

> 3. Add text indentation to attributes in the form of the style padding-left

You're talking about directly adding 'style' to attributes, with
CSS contents?  That would make the docx reader very good for
converting to HTML and not so good for any other format.

Note that in general pandoc does not strive to preserve every
small detail of formatting, only structure.  See the beginning
of the manual.

> 4. Add cell shading and text color to attributes in the form of the styles 
> background-color and color

See above, also search the issue tracker for 'color'.

> 5. Add table borders to attributes in the form of the style border

I think this falls into the category of things that are beyond
pandoc's scope.  We don't strive to reproduce all the formatting
details in conversions.  Again, see the beginning of the manual.

Re: Feature Idea: docx -> HTML table styling

[Noah Malmed]

[-- Attachment #1.1: Type: text/plain, Size: 4195 bytes --]

Hi John,

Thanks for responding! I have a few clarifying questions mainly around 
Attr, because I don't quite understand what values are stored.

When you say adding `vertical-align` to attributes should be okay, how do 
you mean? Would it be more appropriate to store it in the XML format that 
docx uses to denote vertical alignment?

As well, I think I was a little thrown off by some of the black box testing 
we did on the HTML reader. When we ran `pandoc -f html -t native` on the 
following HTML:

<table>
   <tbody>
      <tr>
         <td style="background-color: green>green background</td>
      </tr>
   </tbody>
</table>

We received the following output:

[ Table
    ( "" , [] , [] )
    (Caption Nothing [])
    [ ( AlignDefault , ColWidthDefault ) ]
    (TableHead ( "" , [] , [] ) [])
    [ TableBody
        ( "" , [] , [] )
        (RowHeadColumns 0)
        []
        [ Row
            ( "" , [] , [] )
            [ Cell
                ( "" , [] , [ ( "style" , "background-color: green" ) ] )
                AlignDefault
                (RowSpan 1)
                (ColSpan 1)
                [ Plain [ Str "green" , Space , Str "background" ] ]
            ]
        ]
    ]
    (TableFoot ( "" , [] , [] ) [])
]

Seeing that the style was preserved led us to believe that it would be 
appropriate to store some styling in the AST. Is the problem with our 
proposed solution that we would be storing information that would be 
specific to HTML? Is there maybe a more generic language that would be more 
appropriate to store that information in?  

Thanks,

Noah
 


On Wednesday, June 15, 2022 at 5:36:18 PM UTC-5 John MacFarlane wrote:

> Noah Malmed <nma...-O2gogPphfo5dNrB6XyqITwC/G2K4zDHf@public.gmane.org> writes:
>
> > Hello!
> >
> > We use Pandoc often to convert from docx to HTML, and many of the 
> > documents we convert include tables. As far as we can tell, almost all 
> of 
> > the table styling is lost in the docx reader. Specifically, we care 
> about 5 
> > things:
> >
> > 1. Text justification (left, center, or right)
> >
> > 2. Vertical alignment (top, middle, or bottom)
> >
> > 3. Text indentation
> >
> > 4. Cell shading and text color
> >
> > 5. Table borders 
> >
> > We hope to enhance the docx reader so that these stylings get preserved 
> in 
> > the AST.
> >
> > Proposed solutions:
> >
> > 1. It seems like text justification already exists in the AST through 
> the 
> > Alignment value. It just needs to get implemented in the docx reader, as 
> > described in this issue: https://github.com/jgm/pandoc/issues/6316
>
> Correct.
>
> > 2. Add the vertical alignment style to attributes as suggested here 
> > <https://github.com/jgm/pandoc/issues/7444#issuecomment-881649125>
>
> Should be okay. However, adding `vertical-align` there won't do
> any good for converting to HTML unless the HTML writer is
> modified to be sensitive to this attribute.
>
> > 3. Add text indentation to attributes in the form of the style 
> padding-left
>
> You're talking about directly adding 'style' to attributes, with
> CSS contents? That would make the docx reader very good for
> converting to HTML and not so good for any other format.
>
> Note that in general pandoc does not strive to preserve every
> small detail of formatting, only structure. See the beginning
> of the manual.
>
> > 4. Add cell shading and text color to attributes in the form of the 
> styles 
> > background-color and color
>
> See above, also search the issue tracker for 'color'.
>
> > 5. Add table borders to attributes in the form of the style border
>
> I think this falls into the category of things that are beyond
> pandoc's scope. We don't strive to reproduce all the formatting
> details in conversions. Again, see the beginning of the manual.
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ec31e976-089b-4916-949a-fad874b2a8adn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 7137 bytes --]

Re: Feature Idea: docx -> HTML table styling

[John MacFarlane]

Noah Malmed <nmalmed-O2gogPphfo5dNrB6XyqITwC/G2K4zDHf@public.gmane.org> writes:

> Hi John,
>
> Thanks for responding! I have a few clarifying questions mainly around 
> Attr, because I don't quite understand what values are stored.
>
> When you say adding `vertical-align` to attributes should be okay, how do 
> you mean? Would it be more appropriate to store it in the XML format that 
> docx uses to denote vertical alignment?

I would say not.  We'd prefer something generic that could be
interpreted by various writers, not something docx-specific.

> As well, I think I was a little thrown off by some of the black box testing 
> we did on the HTML reader. When we ran `pandoc -f html -t native` on the 

Yes, it's true that the HTML reader will store style attributes
(and any other attributes) on table cells.  These won't really
have much meaning outside of an HTML context.  But it makes more
sense for the HTML reader to behave this way than, e.g., for
the docx reader to add HTML-specific attributes.

I don't know what the best solution is.  Pandoc is really
designed to preserve the features of documents that are encoded
in its types.  Anything else  -- including the use of
"stringly typed" attributes -- is a bit of a hack, and tends to
lead to unmaintainable messes down the road.

We could try to establish some generic conventions for
representing things like vertical-alignment in the attributes,
e.g. with a "vertical-align" attribute that gets interpreted
by different writers, but as I said, once we depart from the
structured types things get to be a bit of a mess.

Re: Feature Idea: docx -> HTML table styling

[Daniel Staal]

On 6/16/22 1:21 PM, John MacFarlane wrote:
> I don't know what the best solution is.  Pandoc is really
> designed to preserve the features of documents that are encoded
> in its types.  Anything else  -- including the use of
> "stringly typed" attributes -- is a bit of a hack, and tends to
> lead to unmaintainable messes down the road.
> 
> We could try to establish some generic conventions for
> representing things like vertical-alignment in the attributes,
> e.g. with a "vertical-align" attribute that gets interpreted
> by different writers, but as I said, once we depart from the
> structured types things get to be a bit of a mess.

A thought: One option would be to suggest some generic conventions for 
readers, but maintain that *writers* will ignore anything not in the 
official AST.  This would allow people like the original poster to 
handle many issues with custom LUA filters/writers, and if any 
particular attribute becomes widespread in readers you can decide if you 
want to adopt it into the official AST and add it to writers.

Of course, it does tend to push Pandoc more to being a platform for 
converters than strictly a converter itself.

Daniel T. Staal

-- 
---------------------------------------------------------------
This email copyright the author.  Unless otherwise noted, you
are expressly allowed to retransmit, quote, or otherwise use
the contents for non-commercial purposes.  This copyright will
expire 5 years after the author's death, or in 30 years,
whichever is longer, unless such a period is in excess of
local copyright law.
---------------------------------------------------------------

Re: Feature Idea: docx -> HTML table styling

[John MacFarlane]

Daniel Staal <DStaal-Jdbf3xiKgS8@public.gmane.org> writes:

> On 6/16/22 1:21 PM, John MacFarlane wrote:
>> I don't know what the best solution is.  Pandoc is really
>> designed to preserve the features of documents that are encoded
>> in its types.  Anything else  -- including the use of
>> "stringly typed" attributes -- is a bit of a hack, and tends to
>> lead to unmaintainable messes down the road.
>> 
>> We could try to establish some generic conventions for
>> representing things like vertical-alignment in the attributes,
>> e.g. with a "vertical-align" attribute that gets interpreted
>> by different writers, but as I said, once we depart from the
>> structured types things get to be a bit of a mess.
>
> A thought: One option would be to suggest some generic conventions for 
> readers, but maintain that *writers* will ignore anything not in the 
> official AST.  This would allow people like the original poster to 
> handle many issues with custom LUA filters/writers, and if any 
> particular attribute becomes widespread in readers you can decide if you 
> want to adopt it into the official AST and add it to writers.

That's more or less what I was thinking.

Re: Feature Idea: docx -> HTML table styling

[ivo....-xwz7R8GQi1g@public.gmane.org]

[-- Attachment #1.1: Type: text/plain, Size: 1891 bytes --]


my use case is to use pandoc md as a quasi-universal way to write text that 
I can output to pdf, html, and pptx.  unfortunately, I am limited by the 
lack of multi-column cells :-(.



On Friday, June 17, 2022 at 10:36:15 AM UTC-7 John MacFarlane wrote:

> Daniel Staal <DSt...-Jdbf3xiKgS8@public.gmane.org> writes:
>
> > On 6/16/22 1:21 PM, John MacFarlane wrote:
> >> I don't know what the best solution is. Pandoc is really
> >> designed to preserve the features of documents that are encoded
> >> in its types. Anything else -- including the use of
> >> "stringly typed" attributes -- is a bit of a hack, and tends to
> >> lead to unmaintainable messes down the road.
> >> 
> >> We could try to establish some generic conventions for
> >> representing things like vertical-alignment in the attributes,
> >> e.g. with a "vertical-align" attribute that gets interpreted
> >> by different writers, but as I said, once we depart from the
> >> structured types things get to be a bit of a mess.
> >
> > A thought: One option would be to suggest some generic conventions for 
> > readers, but maintain that *writers* will ignore anything not in the 
> > official AST. This would allow people like the original poster to 
> > handle many issues with custom LUA filters/writers, and if any 
> > particular attribute becomes widespread in readers you can decide if you 
> > want to adopt it into the official AST and add it to writers.
>
> That's more or less what I was thinking.
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/3dff9bb1-eed9-4252-9b72-1aa090c5865fn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2688 bytes --]

Re: Feature Idea: docx -> HTML table styling

[William Lupton]

[-- Attachment #1: Type: text/plain, Size: 2986 bytes --]

Are you aware of this list-table filter (it supports rowspan and colspan)?
https://github.com/pandoc/lua-filters/tree/master/list-table

I've fixed a probable align/widths bug for which I haven't yet submitted as
a pull request (sorry...). Also, I'm not sure that its pandoc version logic
is quite right. It's been working fine for me with pandoc 2.16.1.

On Tue, 21 Jun 2022 at 17:49, ivo....-xwz7R8GQi1g@public.gmane.org <ivo.welch-xwz7R8GQi1g@public.gmane.org> wrote:

>
> my use case is to use pandoc md as a quasi-universal way to write text
> that I can output to pdf, html, and pptx.  unfortunately, I am limited by
> the lack of multi-column cells :-(.
>
>
>
> On Friday, June 17, 2022 at 10:36:15 AM UTC-7 John MacFarlane wrote:
>
>> Daniel Staal <DSt...-Jdbf3xiKgS8@public.gmane.org> writes:
>>
>> > On 6/16/22 1:21 PM, John MacFarlane wrote:
>> >> I don't know what the best solution is. Pandoc is really
>> >> designed to preserve the features of documents that are encoded
>> >> in its types. Anything else -- including the use of
>> >> "stringly typed" attributes -- is a bit of a hack, and tends to
>> >> lead to unmaintainable messes down the road.
>> >>
>> >> We could try to establish some generic conventions for
>> >> representing things like vertical-alignment in the attributes,
>> >> e.g. with a "vertical-align" attribute that gets interpreted
>> >> by different writers, but as I said, once we depart from the
>> >> structured types things get to be a bit of a mess.
>> >
>> > A thought: One option would be to suggest some generic conventions for
>> > readers, but maintain that *writers* will ignore anything not in the
>> > official AST. This would allow people like the original poster to
>> > handle many issues with custom LUA filters/writers, and if any
>> > particular attribute becomes widespread in readers you can decide if
>> you
>> > want to adopt it into the official AST and add it to writers.
>>
>> That's more or less what I was thinking.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/3dff9bb1-eed9-4252-9b72-1aa090c5865fn%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/3dff9bb1-eed9-4252-9b72-1aa090c5865fn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxgMdDR%3DXZkZLJhPDHdouxwJ%3Djb53mrsn%2B6YRp1MuEC6vw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 4372 bytes --]

Re: Feature Idea: docx -> HTML table styling

[Albert Krewinkel]

Hi Noah,

Just chiming in to report on our experiences with tables in a project
where we used pandoc to publish journal articles. Our main goal there
was to publish HTML and PDFs from Docx inputs, with an option to handle
JATS as well (Project: <https://oa-pub.hos.tuhh.de/en/project/> Journal:
<https://kommunikation-gesellschaft.de>).

We found that authors writing in Word essentially use tables as a
graphic and layouting tool. Markdown was used as our central format,
which worked extremely well: we converted Docx -> Markdown, fixed markup
when necessary, then published via pandoc. Just tables proved
problematic. For some tables, we ended up writing separate HTML and PDFs
by hand. See the "Sonderausgabe | Podcast" in that journal for results.

This is just to say that pandoc may not be the right tool if you aim for
*fully automatic* conversion of scholarly Docx articles. Maybe tables
should just be expected to require manual tuning.

I believe that [transpect](https://transpect.github.io) tries to
preserve more of the styling, maybe it is more in line with what you
need? Citation support isn't as complete though (last I heard).

Happy to answer questions about any of this.

Cheers,
Albert


Noah Malmed <nmalmed-O2gogPphfo5dNrB6XyqITwC/G2K4zDHf@public.gmane.org> writes:

> Hello!
>
>  We use Pandoc often to convert from docx to HTML, and many of the
> documents we convert include tables. As far as we can tell, almost all
> of the table styling is lost in the docx reader. Specifically, we care
> about 5 things:
>
> 1. Text justification (left, center, or right)
>
> 2. Vertical alignment (top, middle, or bottom)
>
> 3. Text indentation
>
> 4. Cell shading and text color
>
> 5. Table borders
>
> We hope to enhance the docx reader so that these stylings get preserved
> in the AST.
>
> Proposed solutions:
>
> 1. It seems like text justification  already exists in the AST through
> the Alignment value. It just needs to get implemented in the docx
> reader, as described in this issue:
> https://github.com/jgm/pandoc/issues/6316
>
> 2. Add the vertical alignment style to attributes as suggested here
>
> 3. Add text indentation to attributes in the form of the style
> padding-left
>
> 4. Add cell shading and text color to attributes in the form of the
> styles background-color and color
>
> 5. Add table borders to attributes in the form of the style border
>
>
> Does this sound like a sane and feasible solution? We're pretty
> motivated and willing to work on these changes, just want to know if
> they would be the best route!


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Feature Idea: docx -> HTML table styling

[Noah Malmed]

[-- Attachment #1.1: Type: text/plain, Size: 3398 bytes --]

Hi Albert!

Thanks for the info! We actually have basically the same use case! 
Definitely will check out transpect!

Thanks,

Noah

On Thursday, June 16, 2022 at 2:21:37 AM UTC-5 Albert Krewinkel wrote:

> Hi Noah,
>
> Just chiming in to report on our experiences with tables in a project
> where we used pandoc to publish journal articles. Our main goal there
> was to publish HTML and PDFs from Docx inputs, with an option to handle
> JATS as well (Project: <https://oa-pub.hos.tuhh.de/en/project/> Journal:
> <https://kommunikation-gesellschaft.de>).
>
> We found that authors writing in Word essentially use tables as a
> graphic and layouting tool. Markdown was used as our central format,
> which worked extremely well: we converted Docx -> Markdown, fixed markup
> when necessary, then published via pandoc. Just tables proved
> problematic. For some tables, we ended up writing separate HTML and PDFs
> by hand. See the "Sonderausgabe | Podcast" in that journal for results.
>
> This is just to say that pandoc may not be the right tool if you aim for
> *fully automatic* conversion of scholarly Docx articles. Maybe tables
> should just be expected to require manual tuning.
>
> I believe that [transpect](https://transpect.github.io) tries to
> preserve more of the styling, maybe it is more in line with what you
> need? Citation support isn't as complete though (last I heard).
>
> Happy to answer questions about any of this.
>
> Cheers,
> Albert
>
>
> Noah Malmed <nma...-O2gogPphfo5dNrB6XyqITwC/G2K4zDHf@public.gmane.org> writes:
>
> > Hello!
> >
> > We use Pandoc often to convert from docx to HTML, and many of the
> > documents we convert include tables. As far as we can tell, almost all
> > of the table styling is lost in the docx reader. Specifically, we care
> > about 5 things:
> >
> > 1. Text justification (left, center, or right)
> >
> > 2. Vertical alignment (top, middle, or bottom)
> >
> > 3. Text indentation
> >
> > 4. Cell shading and text color
> >
> > 5. Table borders
> >
> > We hope to enhance the docx reader so that these stylings get preserved
> > in the AST.
> >
> > Proposed solutions:
> >
> > 1. It seems like text justification already exists in the AST through
> > the Alignment value. It just needs to get implemented in the docx
> > reader, as described in this issue:
> > https://github.com/jgm/pandoc/issues/6316
> >
> > 2. Add the vertical alignment style to attributes as suggested here
> >
> > 3. Add text indentation to attributes in the form of the style
> > padding-left
> >
> > 4. Add cell shading and text color to attributes in the form of the
> > styles background-color and color
> >
> > 5. Add table borders to attributes in the form of the style border
> >
> >
> > Does this sound like a sane and feasible solution? We're pretty
> > motivated and willing to work on these changes, just want to know if
> > they would be the best route!
>
>
> -- 
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ad6d8a3c-2e96-46e6-af3c-7370801f67c6n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 5428 bytes --]