From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.text.pandoc/28437 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: "christi...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org" Newsgroups: gmane.text.pandoc Subject: Re: anchor-checker.php: a simple script to check HTML self-link validity Date: Mon, 24 May 2021 14:04:50 -0700 (PDT) Message-ID: References: Reply-To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_7338_937752377.1621890290185" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33059"; mail-complaints-to="usenet@ciao.gmane.io" To: pandoc-discuss Original-X-From: pandoc-discuss+bncBCS252WXTEIBB45JWCCQMGQEBWSG6SI-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Mon May 24 23:04:54 2021 Return-path: Envelope-to: gtp-pandoc-discuss@m.gmane-mx.org Original-Received: from mail-oo1-f61.google.com ([209.85.161.61]) by ciao.gmane.io with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1llHks-0008LR-0u for gtp-pandoc-discuss@m.gmane-mx.org; Mon, 24 May 2021 23:04:54 +0200 Original-Received: by mail-oo1-f61.google.com with SMTP id j19-20020a4adf530000b029023ce471f589sf1692298oou.11 for ; Mon, 24 May 2021 14:04:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:date:from:to:message-id:in-reply-to:references:subject :mime-version:x-original-sender:reply-to:precedence:mailing-list :list-id:list-post:list-help:list-archive:list-subscribe :list-unsubscribe; bh=Hc3V2i9wdll6Kwe+ELXgrFjgYmRrb4N4ZG3PyLPhsEA=; b=ZUIiEnCfHoysPpAlzdeSOERkT5MjgJCLodAbl1IL+8x4esB3z6B7kA2wpd8cI8SS6Z Tf8cOHRUHkzmBfyzOcy61uoUqAW3x/3aMmIBNXsq6PVLRwTaRFm8i80CJEqsfauFuaFD GxWCATluC9oF8yncjuSDeXMlAxdd6ikdxRcaHBvqK/AFLpx+BpmgdYfgPToqJCBvcJZy P6fGYZ6MlsojQTF5jPie0qDnUZ2KC/s0jfsJOTsIipYT0B94kpcv3Jko6jboYioYyO4/ lxSb81qtbRjFMHKNDepqmSdwxarBAaGzc79/av6F78hJimznXNrrQCAtHKaRMb6jPink 390w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=Hc3V2i9wdll6Kwe+ELXgrFjgYmRrb4N4ZG3PyLPhsEA=; b=Irmr0+OCJ0+cop76eShwSKGrs4tsel0Bl8tvleX+cTrB9KmipbHcWguNK6J89op/lx +m7VwoXbdAFK3nv/oTIYGJ8hqpzGbEQWqIxNIymIg+detoYvXaygjmqkv9C8M46K6uyx C19pb5k99loQH4je+NkNTP+d5ZWKN/Vx/8sqC0wwZli3Fk5sKhTCRECRh7VZj9T1J47d bKk3raMFtrhcVRq39zI+RhQpN3vBir4cDmkF38wK/PbCxJWzaSoWovWFQRmFvsNIOuC0 PbfsYImZgDBoYj1W/Tw5hL/o5a15yvW3/3MCkeXyXX+tRoxt2LY3TMLQCIGjPtz1Knyg vb0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:date:from:to:message-id:in-reply-to :references:subject:mime-version:x-original-sender:reply-to :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=Hc3V2i9wdll6Kwe+ELXgrFjgYmRrb4N4ZG3PyLPhsEA=; b=irnijK4IAvWod2sU+D5eLb37Hi43dNz5SKdAFjBRbB/MZMxdD45ZxpVekjnY/zdwT+ DTWjAPzfTtU3wlkgDv0kApPDwqIsAlGZDodtEXZK/AmaTlvnbCKeIOLDntKfOshfDP2N yVaFd0dGHLWd14kJBRF8gQEaRT7njOPJkRoLuY/pU7kvmjQLL/prYL5tRZGks6zq5qNK e4HsZsUlFuY6hdrMb2xrfIpiLaRtcINZABr9Zq/6vIfno2DK/X8s5TaGXmYRM84cNoA8 KQkb7y0PACpOERf11ueikWnKEmQQJx9SYeth+DNmvXfhzEBJXJTUtv0WDBczT5rXfzEC yO2A== Original-Sender: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org X-Gm-Message-State: AOAM530Hs9XpCYjHRJdS+BKDwqUqsxlXx/nYGvy4Hh56Hwi2f704W33h j817Oz69MHtRrc+2pJMTtgk= X-Google-Smtp-Source: ABdhPJwRpT+xTvPtaILGYqNTkhVbxXdbrZTWyboy4vRTG3Cas59bpnCWrVGgVf+t261YCklx02nVnw== X-Received: by 2002:aca:1718:: with SMTP id j24mr638829oii.81.1621890292530; Mon, 24 May 2021 14:04:52 -0700 (PDT) X-BeenThere: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org Original-Received: by 2002:a4a:e78e:: with SMTP id x14ls452715oov.5.gmail; Mon, 24 May 2021 14:04:51 -0700 (PDT) X-Received: by 2002:a4a:97ed:: with SMTP id x42mr19592568ooi.40.1621890290724; Mon, 24 May 2021 14:04:50 -0700 (PDT) In-Reply-To: X-Original-Sender: christian.kolen-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org Precedence: list Mailing-list: list pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org; contact pandoc-discuss+owners-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org List-ID: X-Google-Group-Id: 1007024079513 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , Xref: news.gmane.io gmane.text.pandoc:28437 Archived-At: ------=_Part_7338_937752377.1621890290185 Content-Type: multipart/alternative; boundary="----=_Part_7339_269269835.1621890290185" ------=_Part_7339_269269835.1621890290185 Content-Type: text/plain; charset="UTF-8" You may find this useful: https://github.com/filiph/linkcheck On Monday, May 24, 2021 at 7:56:48 AM UTC-7 Gwern Branwen wrote: > When writing long complicated Pandoc documents over years, it is easy > for section or anchor self-links to drift and become invalid. > Pandoc/hakyll will not warn you about links being broken, they do not > break in any visible ways or affect validation, and it's surprisingly > difficult to get any linkchecking tool to warn you about them > specifically. (The W3C linkchecking tool has no option to check only > anchors and insists on checking all external links, while > 'linkchecker' has a plugin for this I am told is broken and also no > way to avoid checking all external links - this renders both tools > completely infeasible for regularly checking a website the size of > gwern.net, where a linkchecker run may take several days to finish.) > This is despite being really quite a simple task: get all hrefs > starting with '#', check that an ID corresponds, print out any ones > missing. > > dbohdan wrote a PHP script for me to check gwern.net pages, which has > worked well and exposed at least 20 erroneous anchors I've fixed, and > which is fast enough to include in the site sync script so new errors > will be picked up immediately. This may be useful to other > Pandoc/hakyll users. > > Source: > https://github.com/gwern/gwern.net/blob/master/build/anchor-checker.php > > #! /usr/bin/env php > // Check anchors in HTML files. Only checks anchors local to each document. > // Anchors prefixed with a filename are ignored even if they refer to the > // same file. Anchors with no element with the corresponding fragment ID > // are written to stderr prefixed with the filename. > // > // Usage: anchor-checker.php [FILE]... > // > // To the extent possible under law, D. Bohdan has waived all copyright and > // related or neighboring rights to this work. > // > // Date: 2021-05-24. > // Requirements: PHP 7.x with the standard DOM module. > > error_reporting(E_ALL); > > function main($files) { > $exit_code = 0; > > foreach ($files as $file) { > $bad_anchors = check_file($file); > foreach ($bad_anchors as $a) { > fprintf(STDERR, "%s\t%s\n", $file, $a); > $exit_code = 1; > } > } > > exit($exit_code); > } > > function check_file($file) { > $html = file_get_contents($file); > // An ugly hack to get around missing HTML5 support tripping > up the parser. > $html = preg_replace("//", "", $html); > > if (preg_match("/^\s*$/", $html)) return []; > > $dom = new DOMDocument(); > $dom->loadHTML($html, LIBXML_NOERROR | LIBXML_NOWARNING); > > return check_document($dom); > } > > function check_document($dom) { > $ids = (new DOMXpath($dom))->query("//@id"); > $id_set = array(); > > foreach ($ids as $id) { > $id_set["#" . $id->value] = true; > } > > $bad_anchors = array(); > $hrefs = (new DOMXpath($dom))->query("//a/@href"); > foreach ($hrefs as $href) { > $value = trim($href->value); > > if (substr($value, 0, 1) !== "#") continue; > > if (!array_key_exists($value, $id_set)) { > $bad_anchors[] = $value; > } > } > > return $bad_anchors; > } > > main(array_slice($argv, 1)); > > This can be used as a post-compilation check like > `static/build/anchor-checker.php ./_site/"$HTML"` or what have you. > > -- > gwern > https://www.gwern.net > -- You received this message because you are subscribed to the Google Groups "pandoc-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/dbba54c1-ea6b-4308-b90f-ba17010f8b6dn%40googlegroups.com. ------=_Part_7339_269269835.1621890290185 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable You may find this useful:  https://github.com/filiph/linkcheck

On Monday, May 24, 2021 at 7:56:= 48 AM UTC-7 Gwern Branwen wrote:
When writing long complicated Pandoc documents over yea= rs, it is easy
for section or anchor self-links to drift and become invalid.
Pandoc/hakyll will not warn you about links being broken, they do not
break in any visible ways or affect validation, and it's surprising= ly
difficult to get any linkchecking tool to warn you about them
specifically. (The W3C linkchecking tool has no option to check only
anchors and insists on checking all external links, while
'linkchecker' has a plugin for this I am told is broken and als= o no
way to avoid checking all external links - this renders both tools
completely infeasible for regularly checking a website the size of
gwern.net, where a linkchecker run may take several = days to finish.)
This is despite being really quite a simple task: get all hrefs
starting with '#', check that an ID corresponds, print out any = ones
missing.

dbohdan wrote a PHP script for me to check gwern.net= pages, which has
worked well and exposed at least 20 erroneous anchors I've fixed, a= nd
which is fast enough to include in the site sync script so new errors
will be picked up immediately. This may be useful to other
Pandoc/hakyll users.

Source: https://githu= b.com/gwern/gwern.net/blob/master/build/anchor-checker.php

#! /usr/bin/env php
<?php
// Check anchors in HTML files. Only checks anchors local to each = document.
// Anchors prefixed with a filename are ignored even if they refer = to the
// same file. Anchors with no element with the corresponding fragm= ent ID
// are written to stderr prefixed with the filename.
//
// Usage: anchor-checker.php [FILE]...
//
// To the extent possible under law, D. Bohdan has waived all copyr= ight and
// related or neighboring rights to this work.
//
// Date: 2021-05-24.
// Requirements: PHP 7.x with the standard DOM module.

error_reporting(E_ALL);

function main($files) {
$exit_code =3D 0;

foreach ($files as $file) {
$bad_anchors =3D check_file($file);
foreach ($bad_anchors as $a) {
fprintf(STDERR, "%s\t%s\n", $file, $a);
$exit_code =3D 1;
}
}

exit($exit_code);
}

function check_file($file) {
$html =3D file_get_contents($file);
// An ugly hack to get around missing HTML5 support tripping
up the parser.
$html =3D preg_replace("/<wbr>/", "",= $html);

if (preg_match("/^\s*$/", $html)) return [];

$dom =3D new DOMDocument();
$dom->loadHTML($html, LIBXML_NOERROR | LIBXML_NOWARNING);

return check_document($dom);
}

function check_document($dom) {
$ids =3D (new DOMXpath($dom))->query("//@id");
$id_set =3D array();

foreach ($ids as $id) {
$id_set["#" . $id->value] =3D true;
}

$bad_anchors =3D array();
$hrefs =3D (new DOMXpath($dom))->query("//a/@href"= );
foreach ($hrefs as $href) {
$value =3D trim($href->value);

if (substr($value, 0, 1) !=3D=3D "#") continue;

if (!array_key_exists($value, $id_set)) {
$bad_anchors[] =3D $value;
}
}

return $bad_anchors;
}

main(array_slice($argv, 1));

This can be used as a post-compilation check like
`static/build/anchor-checker.php ./_site/"$HTML"` or what hav= e you.

--=20
gwern
https://www.gwern.net

--
You received this message because you are subscribed to the Google Groups &= quot;pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to pand= oc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To view this discussion on the web visit https://groups.google.com/d= /msgid/pandoc-discuss/dbba54c1-ea6b-4308-b90f-ba17010f8b6dn%40googlegroups.= com.
------=_Part_7339_269269835.1621890290185-- ------=_Part_7338_937752377.1621890290185--