From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.text.pandoc/30496 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Kris Wilk Newsgroups: gmane.text.pandoc Subject: Re: RTF to Markdown questions Date: Wed, 27 Apr 2022 16:51:13 -0700 (PDT) Message-ID: References: Reply-To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_216_529223014.1651103473820" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33837"; mail-complaints-to="usenet@ciao.gmane.io" To: pandoc-discuss Original-X-From: pandoc-discuss+bncBCC5D7WV5UIRB45NU6JQMGQE6CT4J4I-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Thu Apr 28 01:51:19 2022 Return-path: Envelope-to: gtp-pandoc-discuss@m.gmane-mx.org Original-Received: from mail-ot1-f60.google.com ([209.85.210.60]) by ciao.gmane.io with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1njrRG-0008Xw-Pk for gtp-pandoc-discuss@m.gmane-mx.org; Thu, 28 Apr 2022 01:51:18 +0200 Original-Received: by mail-ot1-f60.google.com with SMTP id q12-20020a9d578c000000b006043f12876fsf970622oth.19 for ; Wed, 27 Apr 2022 16:51:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:date:from:to:message-id:in-reply-to:references:subject :mime-version:x-original-sender:reply-to:precedence:mailing-list :list-id:list-post:list-help:list-archive:list-subscribe :list-unsubscribe; bh=P7otwGBD6F95zhQHJ9hnCSCkZUzZCWFTGcR7B77k19A=; b=q0sj3S+5w9m8NqKeM5s7KLPBIUuf+mR/xRS6dapBgZAHdWOBhdDKwqXZwDiQLQdLbP SHyCadVp1uyuCaoc/rZ/PYSTCRZbUIvqEBOJQykuFkP1wMkCONvw0ou93wN5tWfxesWm i3STFgL4zVT9u5nCDPU2Il96Mc3t029xCevPsGE7492CQMLV/R37XHxXDExi+NIyRmaH CAfxoUIKJ025jBsN4nGpxgyNvIL0zalh/d0/GWomHnD0WKPVisS3Wu4XRvaZ+WPNUy2a La1Gfqpa1wdWg+kCd2XHyoSoZ6uFkNCbTT5231apGR+sd6+ou3be0hT4iP3R1DdMxi+o gSDg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=reefnet.ca; s=google; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=P7otwGBD6F95zhQHJ9hnCSCkZUzZCWFTGcR7B77k19A=; b=iahLzWY8iK4rSp6wMTShfVgxBnhUZuoVf8jH7eoykTq42Z+P1Dlhs31SOF0EhnYWeP bFvDifIxvROESUYw56x2xikxg6ROB/EfmYU28EgW096+aK4kLspfH/r2fpEIx3/40Vr3 ctRr7II78D3m1rPZyw7WC0BSvp3bFp/W5ZpVA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:date:from:to:message-id:in-reply-to :references:subject:mime-version:x-original-sender:reply-to :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=P7otwGBD6F95zhQHJ9hnCSCkZUzZCWFTGcR7B77k19A=; b=s7JX84vT7G1K/apbjF/pChgwt8tsTFpJTWAIvNYHEjIgQPJZfW+x0/XVlLx6xIflSn uMUZnyKlDtkxX8Y2UhBqZ7zc3bgSeqajJinCfKWA2vdYTFUiJfi9LucQF3GbQBLuuWB3 RcwNHZXQnNJSqPOJjeID521UJLUpRgryzZj9bADrLd9YJGWOKSiNP9tq7uwkKrJhjhWR C/MV0i6KQk7WVSAOAOT84ZA6nKzmYziL/eQJ8Efp04E8pGAnJuwY//6M+Kuq4F/Zcf9a f4fvewdRH5QLWahHG7ihZmWRAEQZEiDhMNXDJAub6BanJi3wEgBAecf8R4e9D3EISa8l QdLg== Original-Sender: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org X-Gm-Message-State: AOAM532+x/YsqVx+2wGnB3DSWobA5LIFU0DDeSsZXIsSkC9kg8EFyDXq I28I/JPlBLaOjoWvX0lbq5U= X-Google-Smtp-Source: ABdhPJyYwKNgYgqgNBxmwxsVmW4GTJoI81schpPazvtj/XyNMfwlS4yMNddK01wf6lRlEFckVcQklw== X-Received: by 2002:a05:6808:118c:b0:2d4:4194:70db with SMTP id j12-20020a056808118c00b002d4419470dbmr14087389oil.93.1651103477556; Wed, 27 Apr 2022 16:51:17 -0700 (PDT) X-BeenThere: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Original-Received: by 2002:a05:6808:2224:b0:325:2247:7c23 with SMTP id bd36-20020a056808222400b0032522477c23ls5377292oib.3.gmail; Wed, 27 Apr 2022 16:51:14 -0700 (PDT) X-Received: by 2002:a05:6808:148f:b0:325:1f58:9dfc with SMTP id e15-20020a056808148f00b003251f589dfcmr9523130oiw.20.1651103474518; Wed, 27 Apr 2022 16:51:14 -0700 (PDT) In-Reply-To: X-Original-Sender: kris-AwXHIjbJCMCw5LPnMra/2Q@public.gmane.org Precedence: list Mailing-list: list pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org; contact pandoc-discuss+owners-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org List-ID: X-Google-Group-Id: 1007024079513 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , Xref: news.gmane.io gmane.text.pandoc:30496 Archived-At: ------=_Part_216_529223014.1651103473820 Content-Type: multipart/alternative; boundary="----=_Part_217_797673135.1651103473820" ------=_Part_217_797673135.1651103473820 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable In this case, no, any such references/markup other than bold/italic are=20 spurious and must be removed. The aim is to reduce the content to plain=20 text with the simplest markup only, for ingestion into a database that will= =20 be used by a web app. Kris On Wednesday, April 27, 2022 at 4:56:22 PM UTC-4 Bastien Dumont wrote: > Don't you have any cross-reference to it in your RTF file? If you have,= =20 > you may want to keep it. > > Le Wednesday 27 April 2022 =C3=A0 01:23:47PM, John MacFarlane a =C3=A9cri= t : > >=20 > > In pandoc's AST, that is a Span with an id. > > So your filter will need to match on Span instead of underline, > > but otherwise same as the last one. > >=20 > > Kris Wilk writes: > >=20 > > > Follow-up question: I just spotted another markup that has been=20 > preserved=20 > > > that I don't recognize (and don't want...only bold/italics). Here's a= =20 > > > snippet of converted RTF to markdown: > > > > > > ...radially outwards. [Eyes]{#_Hlk64921583} black, set in a pale... > > > > > > I'm guessing that's some kind of RTF attribute/tag? I presume it can= =20 > be=20 > > > stripped as easily as the underlines but I have no idea what it is in= =20 > the=20 > > > first place. I suspect the answer is found somewhere in > > > > > > https://pandoc.org/lua-filters.html > > > > > > but I'm not sure where to look. > > > > > > Kris > > > > > > On Wednesday, April 27, 2022 at 2:38:07 PM UTC-4 Kris Wilk wrote: > > > > > >> Thanks, the script to strip the underlines is exactly what I needed.= =20 > As=20 > > >> for the spaces in the bolds/italics, I'll add an issue for that.=20 > Obviously,=20 > > >> this is not pandoc's fault but if you have a workaround that can be= =20 > ported=20 > > >> to the RTF reader at some point, that would be super. > > >> > > >> In the meantime, I guess I'll investigate doing the job myself in=20 > Lua.=20 > > >> Even if it takes me a couple of days to figure out it'll be faster= =20 > than=20 > > >> processing every file manually! > > >> > > >> Kris > > >> > > >> On Wednesday, April 27, 2022 at 2:28:20 PM UTC-4 John MacFarlane=20 > wrote: > > >> > > >>> > > >>> The issue with bold is probably because the RTF file includes > > >>> some spaces inside the boldface emphasis. That is depressingly > > >>> common in word processing documents, and we have code in the docx > > >>> reader, if I recall, that handles it by converting > > >>> > > >>> helloSPACE > > >>> to > > >>> helloSPACE > > >>> > > >>> We could port this over to the RTF reader, I think -- can you > > >>> put up an issue on the tracker so we don't forget? > > >>> > > >>> The other issue can be handled using a simple Lua filter. > > >>> Save it as ununderline.lua and use -L ununderline.lua on > > >>> the command line: > > >>> > > >>> function Underline(el) > > >>> return el.content > > >>> end > > >>> > > >>> You could probably handle the spacing issue with a more complex > > >>> Lua filter, as well. > > >>> > > >>> Kris Wilk writes: > > >>> > > >>> > Sorry if anyone gets this twice, had to correct my formatting... > > >>> > > > >>> > I'm trying to use pandoc (for the first time) to convert some RTF= =20 > files=20 > > >>> to=20 > > >>> > markdown. My goal is to extract the text with ***bold*** and=20 > > >>> **italics**=20 > > >>> > preserved and no other formatting. > > >>> > > > >>> > Simply converting with "pandoc in.rtf -o out.md" produces a=20 > markdown=20 > > >>> file=20 > > >>> > that's not quite what I need. For instance, here's a line from th= e=20 > > >>> output: > > >>> > > > >>> > **[Scientific Name]{.underline}: ***Aplysia parvula *Morch, 1863 > > >>> > > > >>> > FIRST and foremost, pandoc tries to preserve the underlined text,= =20 > which=20 > > >>> I=20 > > >>> > don't want. Can this be disabled? I've tried the "bracketed_spans= "=20 > and " > > >>> > native_spans" extensions but this still processes the underlines= =20 > as: > > >>> > > > >>> > **Scientific Name: ***Aplysia parvula *Morch, 1863 > > >>> > > > >>> > SECOND, at least when I view this in VSCode's markdown preview,= =20 > the=20 > > >>> bold=20 > > >>> > and emphasis are not presented correctly, I guess because they=20 > touch=20 > > >>> each=20 > > >>> > other or have spaces (or both?)? It displays correctly if it's: > > >>> > > > >>> > **Scientific Name:** *Aplysia parvula* Morch, 1863 > > >>> > > > >>> > I realize that the text in the RTF might have the bold/italic=20 > tagged=20 > > >>> > weirdly but is there a way to deal with this or am I just stuck? = I=20 > have=20 > > >>> > about 500 such files to process, so I'm looking for automated=20 > methods. > > >>> > > > >>> > Thanks in advance for any help you can provide! > > >>> > > > >>> > --=20 > > >>> > You received this message because you are subscribed to the Googl= e=20 > > >>> Groups "pandoc-discuss" group. > > >>> > To unsubscribe from this group and stop receiving emails from it,= =20 > send=20 > > >>> an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > > >>> > To view this discussion on the web visit=20 > > >>>=20 > https://groups.google.com/d/msgid/pandoc-discuss/aecd40a2-09db-4e1b-96ad-= 752973375e0cn%40googlegroups.com > > >>> . > > >>> > > >> > > > > > > --=20 > > > You received this message because you are subscribed to the Google=20 > Groups "pandoc-discuss" group. > > > To unsubscribe from this group and stop receiving emails from it, sen= d=20 > an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > > > To view this discussion on the web visit=20 > https://groups.google.com/d/msgid/pandoc-discuss/dec1524b-c96f-4090-be4c-= 6a2509879d59n%40googlegroups.com > . > >=20 > > --=20 > > You received this message because you are subscribed to the Google=20 > Groups "pandoc-discuss" group. > > To unsubscribe from this group and stop receiving emails from it, send= =20 > an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > > To view this discussion on the web visit=20 > https://groups.google.com/d/msgid/pandoc-discuss/yh480kbkwmtgvg.fsf%40joh= nmacfarlane.net > . > --=20 You received this message because you are subscribed to the Google Groups "= pandoc-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/= pandoc-discuss/f35cb187-3a19-444b-a9ce-9799e816c305n%40googlegroups.com. ------=_Part_217_797673135.1651103473820 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable In this case, no, any such references/markup other than bold/italic are spu= rious and must be removed. The aim is to reduce the content to plain text w= ith the simplest markup only, for ingestion into a database that will be us= ed by a web app.

Kris

On Wednesday, April 27, 2022 a= t 4:56:22 PM UTC-4 Bastien Dumont wrote:
Don't you have any cross-reference to it in= your RTF file? If you have, you may want to keep it.

Le Wednesday 27 April 2022 =C3=A0 01:23:47PM, John MacFarlane a =C3=A9c= rit :
>=20
> In pandoc's AST, that is a Span with an id.
> So your filter will need to match on Span instead of underline,
> but otherwise same as the last one.
>=20
> Kris Wilk <kr...@ree= fnet.ca> writes:
>=20
> > Follow-up question: I just spotted another markup that has be= en preserved=20
> > that I don't recognize (and don't want...only bold/it= alics). Here's a=20
> > snippet of converted RTF to markdown:
> >
> > ...radially outwards. [Eyes]{#_Hlk64921583} black, set in a p= ale...
> >
> > I'm guessing that's some kind of RTF attribute/tag? I= presume it can be=20
> > stripped as easily as the underlines but I have no idea what = it is in the=20
> > first place. I suspect the answer is found somewhere in
> >
> > https://pandoc= .org/lua-filters.html
> >
> > but I'm not sure where to look.
> >
> > Kris
> >
> > On Wednesday, April 27, 2022 at 2:38:07 PM UTC-4 Kris Wilk wr= ote:
> >
> >> Thanks, the script to strip the underlines is exactly wha= t I needed. As=20
> >> for the spaces in the bolds/italics, I'll add an issu= e for that. Obviously,=20
> >> this is not pandoc's fault but if you have a workarou= nd that can be ported=20
> >> to the RTF reader at some point, that would be super.
> >>
> >> In the meantime, I guess I'll investigate doing the j= ob myself in Lua.=20
> >> Even if it takes me a couple of days to figure out it'= ;ll be faster than=20
> >> processing every file manually!
> >>
> >> Kris
> >>
> >> On Wednesday, April 27, 2022 at 2:28:20 PM UTC-4 John Mac= Farlane wrote:
> >>
> >>>
> >>> The issue with bold is probably because the RTF file = includes
> >>> some spaces inside the boldface emphasis. That is dep= ressingly
> >>> common in word processing documents, and we have code= in the docx
> >>> reader, if I recall, that handles it by converting
> >>>
> >>> <b>helloSPACE</b>
> >>> to
> >>> <b>hello</b>SPACE
> >>>
> >>> We could port this over to the RTF reader, I think --= can you
> >>> put up an issue on the tracker so we don't forget= ?
> >>>
> >>> The other issue can be handled using a simple Lua fil= ter.
> >>> Save it as ununderline.lua and use -L ununderline.lua= on
> >>> the command line:
> >>>
> >>> function Underline(el)
> >>> return el.content
> >>> end
> >>>
> >>> You could probably handle the spacing issue with a mo= re complex
> >>> Lua filter, as well.
> >>>
> >>> Kris Wilk <kr...-AwXHIjbJCMCw5LPnMra/2Q@public.gmane.org> writes:
> >>>
> >>> > Sorry if anyone gets this twice, had to correct = my formatting...
> >>> >
> >>> > I'm trying to use pandoc (for the first time= ) to convert some RTF files=20
> >>> to=20
> >>> > markdown. My goal is to extract the text with **= *bold*** and=20
> >>> **italics**=20
> >>> > preserved and no other formatting.
> >>> >
> >>> > Simply converting with "pandoc in.rtf -o ou= t.md" produces a markdown=20
> >>> file=20
> >>> > that's not quite what I need. For instance, = here's a line from the=20
> >>> output:
> >>> >
> >>> > **[Scientific Name]{.underline}: ***Aplysia parv= ula *Morch, 1863
> >>> >
> >>> > FIRST and foremost, pandoc tries to preserve the= underlined text, which=20
> >>> I=20
> >>> > don't want. Can this be disabled? I've t= ried the "bracketed_spans" and "
> >>> > native_spans" extensions but this still pro= cesses the underlines as:
> >>> >
> >>> > **<u>Scientific Name</u>: ***Aplysia= parvula *Morch, 1863
> >>> >
> >>> > SECOND, at least when I view this in VSCode'= s markdown preview, the=20
> >>> bold=20
> >>> > and emphasis are not presented correctly, I gues= s because they touch=20
> >>> each=20
> >>> > other or have spaces (or both?)? It displays cor= rectly if it's:
> >>> >
> >>> > **Scientific Name:** *Aplysia parvula* Morch, 18= 63
> >>> >
> >>> > I realize that the text in the RTF might have th= e bold/italic tagged=20
> >>> > weirdly but is there a way to deal with this or = am I just stuck? I have=20
> >>> > about 500 such files to process, so I'm look= ing for automated methods.
> >>> >
> >>> > Thanks in advance for any help you can provide!
> >>> >
> >>> > --=20
> >>> > You received this message because you are subscr= ibed to the Google=20
> >>> Groups "pandoc-discuss" group.
> >>> > To unsubscribe from this group and stop receivin= g emails from it, send=20
> >>> an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
> >>> > To view this discussion on the web visit=20
> >>> https://groups= .google.com/d/msgid/pandoc-discuss/aecd40a2-09db-4e1b-96ad-752973375e0cn%40= googlegroups.com
> >>> .
> >>>
> >>
> >
> > --=20
> > You received this message because you are subscribed to the G= oogle Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from= it, send an email to pandoc-dis= cus...-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/dec1524= b-c96f-4090-be4c-6a2509879d59n%40googlegroups.com.
>=20
> --=20
> You received this message because you are subscribed to the Google= Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, = send an email to pandoc-discus..= .@googlegroups.com.
> To view this discussion on the web visit https://groups.googl= e.com/d/msgid/pandoc-discuss/yh480kbkwmtgvg.fsf%40johnmacfarlane.net.

--
You received this message because you are subscribed to the Google Groups &= quot;pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to pand= oc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To view this discussion on the web visit https://groups.google.com/d= /msgid/pandoc-discuss/f35cb187-3a19-444b-a9ce-9799e816c305n%40googlegroups.= com.
------=_Part_217_797673135.1651103473820-- ------=_Part_216_529223014.1651103473820--