From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.text.pandoc/22069 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Simon Michael Newsgroups: gmane.text.pandoc Subject: Re: lua filter to process github wiki links ? Date: Sun, 10 Feb 2019 10:25:37 -0800 Message-ID: <3F388E0A-32D1-4B0D-B7CB-EBDBCFCFA42D@joyful.com> References: Reply-To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Mime-Version: 1.0 (Mac OS X Mail 12.0 \(3445.100.39\)) Content-Type: multipart/alternative; boundary="Apple-Mail=_B1D02A28-4929-4AE1-8F90-56EAC44A232E" Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="185311"; mail-complaints-to="usenet@blaine.gmane.org" To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Original-X-From: pandoc-discuss+bncBDAO3OPDQ4OBBJWZQHRQKGQEGMKQ5FA-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Sun Feb 10 19:25:47 2019 Return-path: Envelope-to: gtp-pandoc-discuss@m.gmane.org Original-Received: from mail-pl1-f183.google.com ([209.85.214.183]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1gstnV-000m3k-7p for gtp-pandoc-discuss@m.gmane.org; Sun, 10 Feb 2019 19:25:45 +0100 Original-Received: by mail-pl1-f183.google.com with SMTP id v2sf7532834plg.6 for ; Sun, 10 Feb 2019 10:25:44 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1549823143; cv=pass; d=google.com; s=arc-20160816; b=ZH5P1tfj62GgNM2evzbpU7yMrfhAY0uUd6og7egpUfG6xYmOlqru40zndjmu6uXsVz SyeDbIxbx6tkKcK/wekW47zmBtI291DOAC1LuyUsOiTU8MpGSsvWHvTTkAQpGLIkGLGT nHkd7vv46HiNldSsUPkd6GMWnOX3Zq74+B9Z37wjUOD/0aBFHbKjmkg7TJm8ZxdojJGY PWqUpiJ5BrpXTT56OALRvwWB0sgzigecfb2Z9OTg9fyxLXevov0U7QetXmcTIZtSNngF tQ5wn69Op1Iot05hQhx4Fi+1khkgyDT1OoIS77wW5WUqnvCLOS3zyeOucd6zqL60kcun bHig== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:message-id:in-reply-to:to :references:date:subject:mime-version:from:sender:dkim-signature; bh=WvqNM1KTQ23BzmG8hKugvw5es27mdQTW/B7ESBQ4HiI=; b=K2UhORCN6ZJJTnO8REuynXfRTNB/3kYK26s2oEfKbK1ywNrT2WGhlbq0K0WteeBAZO EK6njqeE7xFLgtHm+V3qQ50c0D7cxAT+4UEez6WOn0H1JzwydC3UdP5TYK2bRNSZF1EW htR2DYD48vxMBG4XETj9/ZyO12o6MeKSkWrZNKMxgRJmav8cDPXdQbHwXruVDcg+/iox 6PTrMwyPxjjeSj0RwDSnxS41U2CQ+wU+xgmTidzpn+HwuzYhpndQfqfnffP5Ow3Fn3lb kKUEaw3sNv55Iyex7ppy0suFm1+0ulDkqzVem/3oY5XV8gHwZ5mJ5KtormecBJ0m67JJ fxfQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@joyful.com header.s=mesmtp header.b=TBC274+9; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=77ZEE6mQ; spf=pass (google.com: domain of simon-jbxitMBJ2LLQT0dZR+AlfA@public.gmane.org designates 64.147.123.25 as permitted sender) smtp.mailfrom=simon-jbxitMBJ2LLQT0dZR+AlfA@public.gmane.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:from:mime-version:subject:date:references:to:in-reply-to :message-id:x-original-sender:x-original-authentication-results :reply-to:precedence:mailing-list:list-id:list-post:list-help :list-archive:list-subscribe:list-unsubscribe; bh=WvqNM1KTQ23BzmG8hKugvw5es27mdQTW/B7ESBQ4HiI=; b=QjYVvBO/IaD1NkJjDX7OvwpLGkHU6YFz1xMT28CXqpydyYBKL5HFri47/VJrkezBBy WAOv146h7Hvv4i92BQJ/Xo1YuK1/huBOS4OTC1d1Z5SaCTUV5DGhn8sGi5sszpfIXrz/ Wz5PAXpdGA394q4G9lHdarv5Q0htAlpAnF+43noV1Y9JhwD+yDdYKWk8VSLhhITPrmRP XeLJ/XRBLt7bEvphEZNST5UbA6qt5WF+O/VKl5VQvA86e/clCfe3+Hv5N953GLAuroNJ CDLcuOPUmbXwowmTDn8N7tJbrSZ9VUvA526ijmo6k/xYmNxtqspENA+d0x4I1Yy+QlV7 NbVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:from:mime-version:subject:date:references :to:in-reply-to:message-id:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=WvqNM1KTQ23BzmG8hKugvw5es27mdQTW/B7ESBQ4HiI=; b=nfpp1XDiYHMB2KotWkwCq5wIz6GuwhZxSdOQoyUnbI/OhB2gnnvgVlS4dsKL2g02Bp 1liZXn/zjW1lbpEZN4rIwWAlswDVrbqeNju9uuIvlUzDLLZulJDP3JQQb5LJRaVgcO7W cRcLgeqx5TCwJSJSf/iK3k3dJEdpkZvAJzES02GSmOfnJIBuyn0OdNvvox0jwiQQM8iu Q11wiNuoPg0vZhBpgqm9R+07lOU7Wn+yN0cah2bNN/OcAQ7LK4H0AqBNa30RtBEYnM93 Km5tW+JWP7Sl1edhwlPdnZxNHvPF1VRmSQUtG3h7vzVbMfxvk/+NNIEmC4Eb5/dNGYVI Nc+w== Original-Sender: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org X-Gm-Message-State: AHQUAubQbIFpXf56xP5RHJ3c3CtLunlsk0mK1LXxeGFE3jZBzR9dAuYv 0FLhrZscie+mCrIi994+o5M= X-Google-Smtp-Source: AHgI3IZPE6D3hpJd3gW2ogWta3PiNmpudkC2FJ7xsx1haCfu0ULKJIgLKAIpYziRmVueanQ7fZhy7w== X-Received: by 2002:aa7:87d3:: with SMTP id i19mr8590pfo.1.1549823143367; Sun, 10 Feb 2019 10:25:43 -0800 (PST) X-BeenThere: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Original-Received: by 2002:a62:ca47:: with SMTP id n68ls5228071pfg.0.gmail; Sun, 10 Feb 2019 10:25:42 -0800 (PST) X-Received: by 2002:aa7:83d0:: with SMTP id j16mr11443534pfn.92.1549823142544; Sun, 10 Feb 2019 10:25:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1549823142; cv=none; d=google.com; s=arc-20160816; b=vcVSx++3BHmdwQhafkf2DBiS2h7nsR7XyXZZLOccwdH1M1qCuIVO/REkusgqfnRVw3 i/+rVTM/dPSQh2RMQ46vo5C8CZGxMYhbz43gILTwSMhmfuoRKFA7XBpBVogwtlejqPPz Neokg946A6mFe0XX3vYRI2vH3yvuxQuRb+eGGcjRQ1tH1dQmeVIblVX/vc4PeozCnDfw cxzUeNQcOjgksIvf0QFU0+3qpeiwSUmihowYwNuO58rlCHwawEvRhqQcSprM68l/tZ4l GFRIPnH14/ktQjmAV2oaFyxmlEaa3kymIt7rFZVzUk1SKwabIFvXMfj64+N8lVg8biW3 fXuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:in-reply-to:to:references:date:subject:mime-version:from :dkim-signature:dkim-signature; bh=Pa+I3u/wILw9FUn6uyXmRIFVxqwx4KjW2mzlgy3cLHE=; b=HBZ+BMcxMvNcFtlQbGtuiiH6RhS6iIi4Lvfo7oX+tsW/mazlHdEsPUoRWCS4Ji04XZ ECxBVchaJjcByPEn2z7q4F4+tjQCQRAT42uROp0OECnXQjGBanaxZcT02I4C667Gym0g IUafpsV5+jir+pTMf6wlQnScYyt8SBDfrdba/CI2cCP73ngBv9mvNhOXH+1zERDPDwUv cPVAjsPxcBYK0Gcl1VEujUfDlWcfqFmdtfylGrLSSgwha9Ibg39e68hZKCVwa9vN19+K EQtSDcJtwc0q1b5dXvivtLf+zltOZBDmk0sZsItsJ1MQbcwMvFyATaoZBZcNP8BRESIP eGuQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@joyful.com header.s=mesmtp header.b=TBC274+9; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=77ZEE6mQ; spf=pass (google.com: domain of simon-jbxitMBJ2LLQT0dZR+AlfA@public.gmane.org designates 64.147.123.25 as permitted sender) smtp.mailfrom=simon-jbxitMBJ2LLQT0dZR+AlfA@public.gmane.org Original-Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com. [64.147.123.25]) by gmr-mx.google.com with ESMTPS id t14si463688pgv.3.2019.02.10.10.25.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Feb 2019 10:25:42 -0800 (PST) Received-SPF: pass (google.com: domain of simon-jbxitMBJ2LLQT0dZR+AlfA@public.gmane.org designates 64.147.123.25 as permitted sender) client-ip=64.147.123.25; Original-Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 8A3A62E28 for ; Sun, 10 Feb 2019 13:25:40 -0500 (EST) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Sun, 10 Feb 2019 13:25:40 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedtledrleeigdduudefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfquhhtnecuuegrihhlohhuthemucef tddtnecunecujfgurhephfgtggfuffhfvfgjkffosegrtdhmrehhtddvnecuhfhrohhmpe fuihhmohhnucfoihgthhgrvghluceoshhimhhonhesjhhohihfuhhlrdgtohhmqeenucff ohhmrghinhephhhlvggughgvrhdrohhrghdpphgrnhguohgtrdhsihhtvgenucfkphepje eirddujeehrdejgedrudefieenucfrrghrrghmpehmrghilhhfrhhomhepshhimhhonhes jhhohihfuhhlrdgtohhmnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Original-Received: from [192.168.0.6] (cpe-76-175-74-136.socal.res.rr.com [76.175.74.136]) by mail.messagingengine.com (Postfix) with ESMTPA id 591A21030F for ; Sun, 10 Feb 2019 13:25:39 -0500 (EST) In-Reply-To: X-Mailer: Apple Mail (2.3445.100.39) X-Original-Sender: simon-jbxitMBJ2LLQT0dZR+AlfA@public.gmane.org X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@joyful.com header.s=mesmtp header.b=TBC274+9; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=77ZEE6mQ; spf=pass (google.com: domain of simon-jbxitMBJ2LLQT0dZR+AlfA@public.gmane.org designates 64.147.123.25 as permitted sender) smtp.mailfrom=simon-jbxitMBJ2LLQT0dZR+AlfA@public.gmane.org Precedence: list Mailing-list: list pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org; contact pandoc-discuss+owners-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org List-ID: X-Google-Group-Id: 1007024079513 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , Xref: news.gmane.org gmane.text.pandoc:22069 Archived-At: --Apple-Mail=_B1D02A28-4929-4AE1-8F90-56EAC44A232E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" > On Feb 8, 2019, at 3:39 PM, BP Jonsson wrote: >=20 > Den 2019-02-08 kl. 16:47, skrev Simon Michael: >> Hi all, >> I'm working towards using github wiki as a content management system, bu= t rendering the wiki pages myself for better integration with the rest of t= he website. Github's wiki links look like [[Page]] or [[Link text|Page]]. I= can convert them with sed or haskell, but I'm already using pandoc's excel= lent lua filter feature, and I'd like to do the wiki linking that way too i= f feasible. Would anyone have a script for this, or advice on how to tackle= it ? >=20 > In my experience it is actually easier to filter the source to convert wi= ki links to Pandoc links. You just have to take care that you don't do the= change inside code spans and code blocks --- typically by capturing code a= t a higher priority than actual wiki links and just pass them through. >=20 > To convert wiki links to Pandoc links I actually use this: Thanks for this. I agree, this kind of transformation seems more easily don= e at the text level than at the pandoc AST level, in the typical case where= absolute robustness is not needed. I wonder why pandoc's native AST breaks= up strings at spaces. Your scripts are quite interesting, and worth further study for me. Since I= want to be cross-platform and reduce external dependencies, and I'm in a h= askell shake script, I have moved the wiki linking into haskell. For intere= st, here's my current shake rule for generating html from markdown: -- render one website page (from main or wiki repos) as html, -- and save in sites/_site/. -- We first preprocess wiki content (linking github wikilinks etc.) -- in haskell, and then convert to html by running pandoc. "site/_site//*.html" %> \out -> do let filename =3D takeBaseName out pagename =3D fileNameToPageName filename iswikipage =3D filename `elem` wikipagefilenames source | iswikipage =3D "wiki" filename <.> "md" | otherwise =3D "site" filename <.> "md" template =3D "site/site.tmpl" siteRoot =3D if "site/_site/doc//*" ?=3D=3D out then "../.." el= se "." need [source, template] -- read markdown source, link any wikilinks, pipe it to pandoc,=20 -- write html out Stdin . wikify (if iswikipage then Just (fileNameToPageName filenam= e) else Nothing) <$> (readFile' source) >>=3D (cmd Shell pandoc fromsrcmd "-t html" "--template" template ("--metadata=3DsiteRoot:" ++ siteRoot) ("--metadata=3D\"title:" ++ pagename ++ "\"") "--lua-filter=3Dtools/pandoc-site.lua" "-o" out ) This is all a bit complicated right now but I think it's paying off in robu= stness and power. It has allowed me to integrate github wiki content in the= http://hledger.org website (see Cookbook), which I'm= very pleased about. Pandoc continues to be a backbone of hledger's docs. T= hanks Pandoc! --=20 You received this message because you are subscribed to the Google Groups "= pandoc-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/= pandoc-discuss/3F388E0A-32D1-4B0D-B7CB-EBDBCFCFA42D%40joyful.com. For more options, visit https://groups.google.com/d/optout. --Apple-Mail=_B1D02A28-4929-4AE1-8F90-56EAC44A232E Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="UTF-8"

On Feb 8, 2019, at 3:39 PM, BP Jonsson <bpjonsson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

Den 2019-02-08 kl. 16:47, sk= rev Simon Michael:
Hi al= l,
I'm working towards using github wiki as a content managem= ent system, but rendering the wiki pages myself for better integration with= the rest of the website. Github's wiki links look like [[Page]] or [[Link = text|Page]]. I can convert them with sed or haskell, but I'm already using = pandoc's excellent lua filter feature, and I'd like to do the wiki linking = that way too if feasible. Would anyone have a script for this, or advice on= how to tackle it ?

In my experie= nce it is actually easier to filter the source to convert wiki links to Pan= doc links.  You just have to take care that you don't do the change in= side code spans and code blocks --- typically by capturing code at a higher= priority than actual wiki links and just pass them through.
=
To convert wiki links to Pandoc links I actually use this:

Thanks for = this. I agree, this kind of transformation seems more easily done at t= he text level than at the pandoc AST level, in the typical case where absol= ute robustness is not needed. I wonder why pandoc's native AST breaks up st= rings at spaces.

Your scripts are quite interesting, and worth further study for m= e. Since I want to be cross-platform and reduce external dependencies, and = I'm in a haskell shake script, I have moved the wiki linking into haskell. = For interest, here's my current shake rule for generating html from markdow= n:

    -- render one website page (from main or wiki repos) = as html,
    -- and save in sites/_site/.
    -- We first preprocess wiki content (linki= ng github wikilinks etc.)
    -- in haskell,= and then convert to html by running pandoc.
  &n= bsp; "site/_site//*.html" %> \out -> do
  &= nbsp;     let filename =3D takeBaseName out
=             pagename =3D fileNameToPageName f= ilename
            iswi= kipage =3D filename `elem` wikipagefilenames
  &n= bsp;         source
    =           | iswikipage =3D "wiki" </> filena= me <.> "md"
          &= nbsp;   | otherwise  =3D "site" </> filename <.> "md"=
            template = =3D "site/site.tmpl"
         = ;   siteRoot =3D if "site/_site/doc//*" ?=3D=3D out then "../.." else = "."
        need [source, template= ]
        -- read markdown source,= link any wikilinks, pipe it to pandoc, 
  &= nbsp;     -- write html out
    &n= bsp;   Stdin . wikify (if iswikipage then Just (fileNameToPageName fil= ename) else Nothing) <$> (readFile' source) >>=3D
          (cmd Shell pandoc fromsrcmd "-t = html"
             =              "--template" template
                &n= bsp;          ("--metadata=3DsiteRoot:" ++ siteRoo= t)
              &n= bsp;            ("--metadata=3D\"title:" ++ p= agename ++ "\"")
          &n= bsp;                "--lua-filter= =3Dtools/pandoc-site.lua"
        =                    "-o" o= ut )

This i= s all a bit complicated right now but I think it's paying off in robustness= and power. It has allowed me to integrate github wiki content in the = http://hledger.org webs= ite (see Cookbook), which I'm very pleased about. Pandoc continues to be a = backbone of hledger's docs. Thanks Pandoc!


--
You received this message because you are subscribed to the Google Groups &= quot;pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to pand= oc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To view this discussion on the web visit https://groups.google.com/d/msgid/= pandoc-discuss/3F388E0A-32D1-4B0D-B7CB-EBDBCFCFA42D%40joyful.com.
For more options, visit http= s://groups.google.com/d/optout.
--Apple-Mail=_B1D02A28-4929-4AE1-8F90-56EAC44A232E--