From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.4 Received: from minnie.tuhs.org (minnie.tuhs.org [50.116.15.146]) by inbox.vuxu.org (Postfix) with ESMTP id 75FB22044E for ; Tue, 21 May 2024 19:56:23 +0200 (CEST) Received: from minnie.tuhs.org (localhost [IPv6:::1]) by minnie.tuhs.org (Postfix) with ESMTP id F066E43AE2; Wed, 22 May 2024 03:56:18 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuhs.org; s=dkim; t=1716314179; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=FMP8HndopesukpyfTSMpKhngtmoeon9BHVHfH5o4CMc=; b=JaTK0yf6eyko5QtZ2tTB32hsTDNIZggJWi0k0i/vjHNSHAGJd9bL0HkpY3gL2S71GH6E8a MaD94IzVn2BZ29mA2O6P5jmyDoQBgqpG3xDe2QHvDOM1GUmBEcD3QDe22znRTzryJyIYNQ 2rC5IWBrKi7TOkiMnD01VjbWIw5hxY0= Received: from mail-4319.protonmail.ch (mail-4319.protonmail.ch [185.70.43.19]) by minnie.tuhs.org (Postfix) with ESMTPS id 1528443ADB for ; Wed, 22 May 2024 03:56:10 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1716314167; x=1716573367; bh=FMP8HndopesukpyfTSMpKhngtmoeon9BHVHfH5o4CMc=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=QYPjrSctQhudRYJ7vDmEV1b6R9VvH+ABfIG7Ga7HqsJMqhgoXUCQbNgIrisSYNmhs BnGqAHtGkCvjI9q932+wt1Icoo6A+9poQU1kY/QRhWEBbTcUeODffy8cIdyk9+dah+ 8gxqswRwmJWQHuVzTOA/4bUyoauRDAhcRGeVaFopXp/K9jlidMQsJjVpuCuGgtxuPv ijhExCcUvmY38sI2IQsqmcXx6Rn/DynxvkGhQWIzXkdTa5l4KtEIFknsVR9kVB/Lpo 9AA9tpm2QxevES47uelRxk/4b5S/j974GwkpDT/CgNGmQfpCkbx+N8pVVk8UXUvEac dJuJev+A7UzLA== Date: Tue, 21 May 2024 17:56:03 +0000 To: The Eunuchs Hysterical Society Message-ID: In-Reply-To: References: <51CC9A0D-122C-4A3D-8BAF-C249489FB817@serissa.com> Feedback-ID: 35591162:user:proton X-Pm-Message-ID: bd6ba183f2975edd2e70dc8deaeed8a90270ab6d MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: 5U5LLGBHKPV4TP5ZW4OM6K5FWK4UOFNH X-Message-ID-Hash: 5U5LLGBHKPV4TP5ZW4OM6K5FWK4UOFNH X-MailFrom: segaloco@protonmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.6b1 Precedence: list Subject: [TUHS] Re: A fuzzy awk. List-Id: The Unix Heritage Society mailing list Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: segaloco via TUHS Reply-To: segaloco On Tuesday, May 21st, 2024 at 9:59 AM, Paul Winalski wrote: > On Tue, May 21, 2024 at 12:09=E2=80=AFAM Serissa wr= ote: >=20 > > > Well this is obviously a hot button topic. AFAIK I was nearby when fu= zz-testing for software was invented. I was the main advocate for hiring An= dy Payne into the Digital Cambridge Research Lab. One of his little project= s was a thing that generated random but correct C programs and fed them to = different compilers or compilers with different switches to see if they cra= shed or generated incorrect results. Overnight, his tester filed 300 or so = bug reports against the Digital C compiler. This was met with substantial p= ushback, but it was a mostly an issue that many of the reports traced to th= e same underlying bugs. > > >=20 > > > Bill McKeemon expanded the technique and published "Differential Test= ing of Software" https://www.cs.swarthmore.edu/~bylvisa1/cs97/f13/Papers/Di= fferentialTestingForSoftware.pdf >=20 > In the mid-late 1980s Bill Mckeeman worked with DEC's compiler product te= ams to introduce fuzz testing into our testing process. As with the C compi= ler work at DEC Cambridge, fuzz testing for other compilers (Fortran, PL/I)= also found large numbers of bugs. >=20 > The pushback from the compiler folks was mainly a matter of priorities. F= uzz testing is very adept at finding edge conditions, but most failing fuzz= tests have syntax that no human programmer would ever write. As a compiler= engineer you have limited time to devote to bug testing. Do you spend that= time addressing real customer issues that have been reported or do you spe= nd it fixing problems with code that no human being would ever write? To ta= ke an example that really happened, a fuzz test consisting of 100 nested pa= rentheses caused an overflow in a parser table (it could only handle 50 nes= ted parens). Is that worth fixing? >=20 > As you pointed out, fuzz test failures tend to occur in clusters and many= of the failures eventually are traced to the same underlying bug. Which le= ads to the counter-argument to the pushback. The fuzz tests are finding rea= l underlying bugs. Why not fix them before a customer runs into them? That = very thing did happen several times. A customer-reported bug was fixed and = suddenly several of the fuzz test problems that had been reported went away= . Another consideration is that, even back in the 1980s, humans weren't the= only ones writing programs. There were programs writing programs and they = sometimes produced bizarre (but syntactically correct) code. >=20 > -Paul W. A happy medium could be including far-out fuzzing to characterize issues, b= ut not necessarily then immediately sink the resources into resolving bizar= re discoveries from the fuzzing. Better to know then not but also have the= wisdom to determine "is someone actually going to trip this" vs. "this is = something that is possible and good to document". In my own work we have s= everal of the latter where something is almost guaranteed to never happen w= ith a human interaction, but is also something we want documented somewhere= so if unlikely problem ever does happen, the discovery is already do= ne and we just start plotting out a solution. That's also some nice low ha= nging fruit to pluck when there isn't much else going on, but avoids the ph= enomenon where we sink critical time into bugfixes with a microscopic ROI. - Matt G.