From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.comp.tex.context/21634 Path: news.gmane.org!not-for-mail From: Mojca Miklavec Newsgroups: gmane.comp.tex.context Subject: Re: Re: test suite in the garden Date: Wed, 27 Jul 2005 00:23:37 +0200 Message-ID: <6faad9f005072615231ef5f138@mail.gmail.com> References: <20050725185214.CC7871A8D6@mendoza.math.indiana.edu> <6faad9f0050725130439c079b2@mail.gmail.com> <3140B5FA-6F65-47E1-AAA0-BD8492696241@fiee.net> Reply-To: Mojca Miklavec , mailing list for ConTeXt users NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Trace: sea.gmane.org 1122416668 27663 80.91.229.2 (26 Jul 2005 22:24:28 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Tue, 26 Jul 2005 22:24:28 +0000 (UTC) Original-X-From: ntg-context-bounces@ntg.nl Wed Jul 27 00:24:26 2005 Return-path: Original-Received: from ronja.vet.uu.nl ([131.211.172.88] helo=ronja.ntg.nl) by ciao.gmane.org with esmtp (Exim 4.43) id 1DxXpy-00056t-7M for gctc-ntg-context-518@m.gmane.org; Wed, 27 Jul 2005 00:23:46 +0200 Original-Received: from localhost (localhost.localdomain [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id D7B9212817; Wed, 27 Jul 2005 00:23:45 +0200 (CEST) Original-Received: from ronja.ntg.nl ([127.0.0.1]) by localhost (smtp.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 23375-10; Wed, 27 Jul 2005 00:23:42 +0200 (CEST) Original-Received: from ronja.vet.uu.nl (localhost.localdomain [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id C8880127EC; Wed, 27 Jul 2005 00:23:41 +0200 (CEST) Original-Received: from localhost (localhost.localdomain [127.0.0.1]) by ronja.ntg.nl (Postfix) with ESMTP id 339B3127EC for ; Wed, 27 Jul 2005 00:23:40 +0200 (CEST) Original-Received: from ronja.ntg.nl ([127.0.0.1]) by localhost (smtp.ntg.nl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 23379-09 for ; Wed, 27 Jul 2005 00:23:38 +0200 (CEST) Original-Received: from nproxy.gmail.com (nproxy.gmail.com [64.233.182.199]) by ronja.ntg.nl (Postfix) with ESMTP id C5159127E9 for ; Wed, 27 Jul 2005 00:23:38 +0200 (CEST) Original-Received: by nproxy.gmail.com with SMTP id a4so10532nfc for ; Tue, 26 Jul 2005 15:23:37 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ZQL0OanOXSRiFLM0CLHL1sYZpvhIoHxn9xzOzbLzt6spqkON2Jsd9uMIy4jQDgRWmo9rYoxT9WReJeC79Mh1xjD+DufH1I5r75MRevg979zGCz9yLUGOBZ1DnNejD+uCWn2lVU1XY6LQOkgMuzR3uuciWYCwjnGCi0VHAByYOw8= Original-Received: by 10.48.142.11 with SMTP id p11mr7811nfd; Tue, 26 Jul 2005 15:23:37 -0700 (PDT) Original-Received: by 10.48.247.2 with HTTP; Tue, 26 Jul 2005 15:23:37 -0700 (PDT) Original-To: mailing list for ConTeXt users In-Reply-To: Content-Disposition: inline X-Virus-Scanned: amavisd-new at ntg.nl X-BeenThere: ntg-context@ntg.nl X-Mailman-Version: 2.1.5 Precedence: list List-Id: mailing list for ConTeXt users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: ntg-context-bounces@ntg.nl Errors-To: ntg-context-bounces@ntg.nl X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on smtp.ntg.nl X-Virus-Scanned: amavisd-new at ntg.nl Xref: news.gmane.org gmane.comp.tex.context:21634 X-Report-Spam: http://spam.gmane.org/gmane.comp.tex.context:21634 Patrick Gundlach wrote: >=20 > >> It is perhaps a lot of work included, but extending the > >> contextgarden.net in such a way that users could provide test cases > >> which would be typeset with different ConTeXt versions and PNGs > >> compared afterwards ... could make it easier to discover any broken > >> functionality. >=20 > It comes down to: >=20 > * different context versions needed Once you are done with the fonts ... Different ConTeXt versions also seem to be useful for all kinds of stuff: - browsing (& comparing) old source files - running live.contextgarden.net on older versions (in case the behaviour in the new version is changed) - (maybe donloading old ConTeXt versions for whatever reason?) - modules.pdf - test suite > * different test documents needed This can be done gradually. Of course, it is impossible to have a very good test suite, but at least some features can be tested; broken functionality would be noted much later otherwise (if ever). If you let the users add stuff, the suit will gradually grow. It would be much easier then to say "take a look: my document is OK with the version this-and-that, but since the version this-and-that something strange happens" when reporting a bug. (Perhaps testing documents should also have at least some very basic set of labels: very important to test, interesting test case, just temporary or one-time testing. Or at least some switch to leave the document there, but to remove it from the list of "test suite" when next ConTeXt version is out. 100 versions and 1000 documents ... can slow down your computer a bit.) > 1) user can select any combination of the above > 2) result (one page/png, more pages pdf) can be viewed or downloaded Converting multiple-page pdf to png-s should also be possible. Take http://archive.contextgarden.net/thread/20050701.172657.13cd3fb5.html for example (the difference seen on the third page). But the number of pages or at least the number of document with a big number of pages has to be limited somehow. > 3) feedback Perhaps some labeling in case the results differ? - Label A: to inspect what's going wrong - Label B: it's ok, it's only a new feature, not supported before - Label C: ok, the bug was removed (some page inbetween differ) - Label D: ... Only the label A and perhaps some others would be interesting then ... > So we need test documents and after that context can mix in the pages > from the known good document and the selecte version. >=20 > So you think that users actually download these comparisons? The most important would be some graphical representation (different colors if a document doesn't compile or if it changes), but allowing the user to download a selected document compiled with a selected version should be possible. > Don't call me magic. I am currently trying to understand the tftopl, > pltotf, vftovp and vptovf programs in detail. And I am so miserable at > that; all kinds of optimizations that makes the code unreadable. :) > > Let me extend the suggestion, for we miss a ConTeXt test suite for a > > long time now: > > > > - users send test cases > > - test cases get typeset with the actual ConTeXt and converted to PNG > > (like inline samples) > > - user "votes" if it looks right, if not adds comment > > - test case gets saved, including PNG, ConTeXt version and vote/comment= s > > - at next update, all test cases get typeset again, and users can > > "vote" if it looks still ok or at least the same (perhaps it would > > be possible to check automatically if the bitmaps are exactly > > identical) >=20 > Again, I doubt that users will actually do these kind of things. I > could provide some interface for documents connected to version > numbers, so you could download a set of .tex files (or one big tex > file) and the related pdf file, but we need to collect good examples. > And I think that testing is very hard: there are so many different > things in ConTeXt that can be tested, so the pdf would result in a few > hundred pages. No, this wouldn't make any sense. I was thinking about PDF to PNG conversion and bitwise comparison of the files, which could be made automatically. Each time a new version would be uploaded, the documents would be compiled, any compilation errors caught and any image differences reported. Only if any of those two cases would be reported, the results would be inspected manually. Sorry Patrick :), Mojca