From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.text.pandoc/25958 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Joseph Reagle Newsgroups: gmane.text.pandoc Subject: Re: WIP: better citation processing Date: Fri, 21 Aug 2020 17:21:52 -0400 Message-ID: <91c2cfed-7211-4194-72cf-5c01abf2315c@reagle.org> References: Reply-To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="14288"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Original-X-From: pandoc-discuss+bncBD65ZAVVYEKRB5HVQD5AKGQECCRFX4I-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Fri Aug 21 23:22:01 2020 Return-path: Envelope-to: gtp-pandoc-discuss@m.gmane-mx.org Original-Received: from mail-pg1-f192.google.com ([209.85.215.192]) by ciao.gmane.io with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1k9EU4-0003YD-MJ for gtp-pandoc-discuss@m.gmane-mx.org; Fri, 21 Aug 2020 23:22:00 +0200 Original-Received: by mail-pg1-f192.google.com with SMTP id j37sf1661177pgi.16 for ; Fri, 21 Aug 2020 14:22:00 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1598044919; cv=pass; d=google.com; s=arc-20160816; b=SRLHw/vjxa45+qtNQbMsAeXTS2ih94ewgdm1DwWMBnfNEwAsSOyegmwdpX4aLA4EQp +gX3deuEmCFd79aZSpOR1ifqpWdff2pqlGxfaldne3EfAoGolzLCyQ8zHS+yv4miuSnU Y7xSmlEgJRTdiCcHjp4hXBFWQCZeQkmCA4ZVdr2jAhhg7O/C1TiyyVTQHwx6k0uGYzFi r6HQ+hchZlb6OJvbF7lLFGrva6pluAWgjm5dXmLvib6RpFZZb0sAmM1UMQ2q0I03TED7 SU01yLgXsB6Hg3lzn2LgCfMt4Ymm61MQmeS8O9UCcMF4Zh2bpWd37wjUX8j33u26UTby SE6A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:to:subject:sender:dkim-signature; bh=pLKc6IecSkOMoY0viWiRqQJRDNYKIOMbVxi3wSuNhMI=; b=Cb9tSTmw+JxWO1vuYU/lw8BTJm6r6lPzTbsaACMjRMIiU+utt+TXu3c6s7X8K+U1So YP916UF2ntTaGxaOMudhzlFUxVF2wgrmz2z95ouet/6wuXEN3oNRAtoVO2hb6OoqzxkR nUogKZ0jA/V82mDefX9sYvSx2zYTeAgdE3D+7QRCCtTlcyb/l0WaxJsDIo6sUdVlIC4F TAKWhIJCIBMD/v/BzY1bwZgOjzrIYikc5r8gOaq0fUeoLaO71jTzE1cDSTi6I4JepfZI ZavywE+w1/h/mnLAL2m1aejoA9hjsRCbcKXczHPpbM8G3eqEuaaECWoAGwdkpsFe1CYc tNWA== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@reagle.org header.s=default header.b=E96d+zys; spf=pass (google.com: domain of joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org designates 23.83.213.36 as permitted sender) smtp.mailfrom=joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:subject:to:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding :x-original-sender:x-original-authentication-results:reply-to :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=pLKc6IecSkOMoY0viWiRqQJRDNYKIOMbVxi3wSuNhMI=; b=iA90kM5eAFMj2SKX4uDOtsugUzsaWVZ+lf8bC07QZafp+UFgXsaffdVJKaM0V8aLjs UqEmyYqUuEvvmZCUqYf5q7B1GLeOgSUDWqNKIwzTIRhdnPj71WWCch+TulAuyBXPgpRF /7JLy4CDrX2zAHS8P8YMh2mzPRjK/fSKN5gDGxlju8j6sy8Qj72iGWxjz4ag2UcGeiT8 hwW6aIQNpr5hqYwZ9W3OYZ9RYEFG9JKpZ5BVsMgdR6Y2TpNri8MB5efST/ChbKoZST7g HZTew3wV77RsgDrFp0p8ZCsGzaQccNUKJr5sL8t3QNUqHi2zGSVri5UXZBz+qecY27l9 MpIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:subject:to:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=pLKc6IecSkOMoY0viWiRqQJRDNYKIOMbVxi3wSuNhMI=; b=HwEdP4MiA4RfJziBjVTlMQCappITtGoS6j3+14eDQtHbgAxMo5Ye3JYyIbz9BjvZjH TGrWM2rrUSjnk4lP54zjou16OHkemI9ztnXIFKqg0UQyORpwKEOZvULhkKW79wz3adnL pjI34Nhv3hspZbEj+sE9hFpWDBmY5RIvs2M2DQASPIig+LUVnuaIp2eagZK9PxEJIVBl fblMaPjQf689SY3BhU/dGzXI3Tyo4sE9j8wGmKHk5Rx5/QraVBe0HIRBT0JcEdb13QV3 ItzUfB+2qKbBKJwf0w+VtnqsZK3AkqSRLYIf Original-Sender: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org X-Gm-Message-State: AOAM533I3tEDqn59W+Uf1F7BMqo/8LvAac464L0m0NIgMhOfOy3PnMG5 NL4TEAA2rNpYEOsJk6NT46k= X-Google-Smtp-Source: ABdhPJy6EVwhzljDtjQTqXVIwPkdwlxxU9DUDu329l7ADeAz6nuE4AN/F97EJAqZf3V4nzIhXtChpw== X-Received: by 2002:a63:1417:: with SMTP id u23mr3672737pgl.289.1598044919397; Fri, 21 Aug 2020 14:21:59 -0700 (PDT) X-BeenThere: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Original-Received: by 2002:a65:620d:: with SMTP id d13ls1058294pgv.0.gmail; Fri, 21 Aug 2020 14:21:56 -0700 (PDT) X-Received: by 2002:a65:5a8d:: with SMTP id c13mr3717882pgt.268.1598044916090; Fri, 21 Aug 2020 14:21:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1598044916; cv=none; d=google.com; s=arc-20160816; b=bKVhGEEvuRXhUbur55ndq/BpcpqI9iFDaStiOBDF8i1s8ANd053IXDfGQjftrBI+hw 8zlUTYsKPXJiKCt83q2/G3kL2F8bEBDq0t82w5WHBriVk0Ue8Dw7LxwLYnuqXUmZxu7t 5kEUMjKXCE6PecAdQIGFQByFKaNzYjCWcRCvExWYPC8Iv3e3+aaOboYtqfOYfY39mT8b VRjr4zoF+5diK3kUoor396Pl8cM0iSEk/VIalaRWCbw7qetVkKiVXEr+U2eeWV/xAJjg NKw85IiW2S2tDoDQ/na8q8upmiHAcRsIlSwQrAQ2pQFIUKwcLmLSO4e+76gvqicve/O6 WsXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :dkim-signature; bh=fqdwPqBgR3TNIU0X1/5B2V3u4hR+IEMI6wNMdfhNkaQ=; b=FdI+cwgo0eC0Z/sR4wM0wjCrtlaqU1W2VT0harVNozpV5vgcsYhfdotfWKPHw5Cj7q UDFIS+68ILHPLfR29L3cqxWvrD4HH0c21q3ms/QmydeQE+NTii67G63GEGvxipO0kkxW A+I5JszlSp/cjNEqF+9YUJJgXDtocg8Ts0RzIdpdnN9bk2HEKSdHaiIvk63z3JjemVKy FjmwFmZcb0AnGL0KZDZe6WMkEuaIRsGFaYMlumXeHAkvTb5j6FS9Icxcy+Vy3RGQc27w bI21h0a6IfMD6koyJwj0B7HZ6Jc/AL4YvLwX+H4hv/fDWvELCdMgsBgq3ZRH0YH6718J UD3g== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@reagle.org header.s=default header.b=E96d+zys; spf=pass (google.com: domain of joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org designates 23.83.213.36 as permitted sender) smtp.mailfrom=joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org Original-Received: from cichlid.larch.relay.mailchannels.net (cichlid.larch.relay.mailchannels.net. [23.83.213.36]) by gmr-mx.google.com with ESMTPS id j4si225068pjd.0.2020.08.21.14.21.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Aug 2020 14:21:55 -0700 (PDT) Received-SPF: pass (google.com: domain of joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org designates 23.83.213.36 as permitted sender) client-ip=23.83.213.36; X-Sender-Id: a2hosting|x-authuser|joseph-T1oY19WcHSwdnm+yROfE0A@public.gmane.org Original-Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id A52CD4814B3 for ; Fri, 21 Aug 2020 21:21:54 +0000 (UTC) Original-Received: from az1-ss21.a2hosting.com (100-96-19-54.trex.outbound.svc.cluster.local [100.96.19.54]) (Authenticated sender: a2hosting) by relay.mailchannels.net (Postfix) with ESMTPA id 0BC1B481242 for ; Fri, 21 Aug 2020 21:21:54 +0000 (UTC) X-Sender-Id: a2hosting|x-authuser|joseph-T1oY19WcHSwdnm+yROfE0A@public.gmane.org Original-Received: from az1-ss21.a2hosting.com (az1-ss21.a2hosting.com [68.66.224.43]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384) by 0.0.0.0:2500 (trex/5.18.8); Fri, 21 Aug 2020 21:21:54 +0000 X-MC-Relay: Good X-MailChannels-SenderId: a2hosting|x-authuser|joseph-T1oY19WcHSwdnm+yROfE0A@public.gmane.org X-MailChannels-Auth-Id: a2hosting X-Minister-Lettuce: 4229a89750b654ed_1598044914335_3467758605 X-MC-Loop-Signature: 1598044914335:1052567101 X-MC-Ingress-Time: 1598044914334 Original-Received: from c-73-149-23-48.hsd1.ma.comcast.net ([73.149.23.48]:65145 helo=[192.168.0.50]) by az1-ss21.a2hosting.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1k9ETx-0008SA-6V for pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org; Fri, 21 Aug 2020 14:21:53 -0700 In-Reply-To: Content-Language: en-US X-AuthUser: joseph-T1oY19WcHSwdnm+yROfE0A@public.gmane.org X-Original-Sender: joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@reagle.org header.s=default header.b=E96d+zys; spf=pass (google.com: domain of joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org designates 23.83.213.36 as permitted sender) smtp.mailfrom=joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org Precedence: list Mailing-list: list pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org; contact pandoc-discuss+owners-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org List-ID: X-Google-Group-Id: 1007024079513 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , Xref: news.gmane.io gmane.text.pandoc:25958 Archived-At: On 8/21/20 3:06 PM, John MacFarlane wrote: > The new library will speed up 2, but it won't affect 1, and I > suspect 1 is the bottleneck for you. I hadn't noticed the performance drop given I continued subsetting my readi= ngs.yaml into a document specific yaml file. > Here's a workaround that should work right now. Reading CSL JSON > is fast, so you could try using pandoc-citeproc -j to convert > your YAML bibliography to CSL, then refer to the CSL bibliography > in your pandoc metadata. It could help, but doesn't practically. Creating a subset of my full bibliography for a document and building a sin= gle ref markdown to HTML takes 1 second, which is great for frequent edits/= builds of my markdown. Using the full YAML bibliography takes 90 seconds.=20 Converting the full bibliography from YAML to JSON with pandoc-citeproc tak= es a similar 90 seconds and then running the simple single-ref document bui= ld with the resulting JSON never exits -- I killed it after four minutes. P= erhaps this is because I typically include the bibliography file as a secon= d input file. (I forget why I do this, there's probably something on this l= ist's archives from years ago.) Using the `--bibliography=3DFILE` approach has much better performance. My = full YAML build takes 35 seconds, and the JSON takes 1 second -- similar to= subsetting the YAML. However, adding 90 seconds to convert from YAML to JS= ON every time I edit my bibliography is a big hit.=20 --=20 You received this message because you are subscribed to the Google Groups "= pandoc-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/= pandoc-discuss/91c2cfed-7211-4194-72cf-5c01abf2315c%40reagle.org.