From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: *** X-Spam-Status: No, score=3.0 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_SBL_CSS autolearn=no autolearn_force=no version=3.4.4 Received: from minnie.tuhs.org (minnie.tuhs.org [IPv6:2600:3c01:e000:146::1]) by inbox.vuxu.org (Postfix) with ESMTP id 14DC221CDD for ; Tue, 21 May 2024 20:12:45 +0200 (CEST) Received: from minnie.tuhs.org (localhost [IPv6:::1]) by minnie.tuhs.org (Postfix) with ESMTP id CF33B43A99; Wed, 22 May 2024 04:12:40 +1000 (AEST) Received: from mout.perfora.net (mout.perfora.net [74.208.4.197]) by minnie.tuhs.org (Postfix) with ESMTPS id 7739E43A99 for ; Wed, 22 May 2024 04:12:32 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=makerlisp.com; s=s1-ionos; t=1716315151; x=1716919951; i=luther.johnson@makerlisp.com; bh=z2GU5ZwP3XBTAOJ9ze/DOnhiPXCaG5ZJG8HaEZLeSOU=; h=X-UI-Sender-Class:Subject:To:References:From:Message-ID:Date: MIME-Version:In-Reply-To:Content-Type:cc: content-transfer-encoding:content-type:date:from:message-id: mime-version:reply-to:subject:to; b=UR4o2r4S/ac9oXR9tjHgokgPguJIo7ShC0M7QeJ/e+PScHJOBb8esuNKpD7tt6n6 WeOVoHlig1bNaDmMfizSLMtrzae12M4DtGvUYuAdV465ksaA/gv/7NjG1xGaKtYYd G8hW/6bNzaYr+LPVHqqhwDQ02lk/b6rOyZtAlI5ftVQoTc1DMUqDU+HHbw7UhAMGL mWPKm+U4/oDQdwMTK4WHVM9Qc/2J5aYgVY66TcXvgcLsa06BbawXzf+aZHy1OJ9Nr fsSnGnRyIsUDDgzDE0f6dUUTreNaaE8HNA4z8usRflsPYCk4/Ri4OR+qrU+oc0MSf c+bmred0zU22GzQK3w== X-UI-Sender-Class: 55c96926-9e95-11ee-ae09-1f7a4046a0f6 Received: from makerlispvps ([74.208.29.250]) by mrelay.perfora.net (mreueus003 [74.208.5.2]) with ESMTPSA (Nemesis) id 0MJRUb-1s87SW2Nqd-0031ZH for ; Tue, 21 May 2024 20:12:31 +0200 Received: from [192.168.234.135] (unknown [172.56.208.41]) by makerlispvps (Postfix) with ESMTPSA id 063488BF41 for ; Tue, 21 May 2024 18:12:30 +0000 (UTC) To: tuhs@tuhs.org References: <51CC9A0D-122C-4A3D-8BAF-C249489FB817@serissa.com> From: Luther Johnson Message-ID: Date: Tue, 21 May 2024 11:12:29 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------1E2455C5EED03ABE5CCD444D" X-Provags-ID: V03:K1:eEdjGFXZQzgb9C56QUlQxKRuk52PQNQpXiHwWPPVzG6WVEsnhv1 S3BVCV2xmksHmhgVTJf8VOaTT6Xxa4YQjUpmBq5wN0mG0PVDN3WX6pkwCrIGxBax8fQGSka NQ2n4/Yuer5WP3nqZRB6i7zlb0F2E7AukQUTX0dUXAipCQOiH5eJ9qBbExm0XpgxyYEbmWK 1qrMckfQLeqOYNx5XtIPQ== UI-OutboundReport: notjunk:1;M01:P0:kLWFwFgmnkE=;y6UR5FIR8hngaMacOqEMb4684YM nSkFsB59WBhasnU2lBnw2eaoD1M9tE6r7xpTOzDnCtvX8cS+dqCv/lsPGkxKD349JZq1BG9n1 hEPwBKtGjbmumaXBTLwWI75nZDI3HwuzfXfxp7NgVd6XxTjP/fZNB2w50qPnvE3g/DrCwEeLf B7b+q8xWcB0xsWZaL/WwkxYPZGNKhPZzJr7O2u9kfncltyoZglGkp2QniYObg8VxFNsAGW3e8 QT5RmaK1bEyrCHEhWoFWyTqpZy83z76+d39YKCJb5Tb8MhlN1Gn1tExFPmU1XVjWo8nWHESZB QDh7ij7wUc6E/xVYbdQJ2mDKDGO4Sk7Sg2E06+BJzq4vR8F4SNOeUj6zB8NHgfLR71d+VD4t4 kAfadI5dRGpVouPc8bhXCfC5XHo7HCqRbd7vy0fkrOyQ12qfMDVJOeaaKEw8uACfThEbb/Qqk rMMNiP8zSt/LWglaZ8djv7npYpek5OtEXYNmNGX7UqXth1bFiD05air7L6iXlKmp7zLdH+Ihh jXZ1utvFNDuFz4s6g8uAFJLWFdaTwvuRDpN/WEQ75iovpjiDnZ3C7DeZkOQqJ13rjiLfeSuWF /7G1vPJOnzd1T2El9G/2tRQUkSDomvc4BYbFQo2Wwlo6PjmfIoPZboDG9p9t3cwDm87FohOkT 9IJFo59OZpxsM2RnwPhUc9UZSFBKVgdrnikRbRvsWkrlRbyTP+novEbvei242r3i2M/LTbsTV 3VT1qhboB50o7y9EYboCluE+XOMiZOx07fmD2gm9eVCPcoD/48aviw= Message-ID-Hash: RON3ZUMBOVJSMMNKUNPWUVFNU4MCVALN X-Message-ID-Hash: RON3ZUMBOVJSMMNKUNPWUVFNU4MCVALN X-MailFrom: luther.johnson@makerlisp.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.6b1 Precedence: list Subject: [TUHS] Re: A fuzzy awk. List-Id: The Unix Heritage Society mailing list Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is a multi-part message in MIME format. --------------1E2455C5EED03ABE5CCD444D Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable I like this anecdote because it points out the difference between being able to handle and process bizarre conditions, as if they were something that should work, which is maybe not that helpful, vs. detecting them and doing something reasonable, like failiing with a "limit exceeded" message. A silent, insidious failure down the line because a limit was exceeded is never good. If "fuzz testing" helps exercise limits and identifies places where software hasn't realized it has exceeded its limits, has run off the end of a table, etc., that seems like a good thing to me. On 05/21/2024 09:59 AM, Paul Winalski wrote: > On Tue, May 21, 2024 at 12:09=E2=80=AFAM Serissa > wrote: > > Well this is obviously a hot button topic. AFAIK I was nearby > when fuzz-testing for software was invented. I was the main > advocate for hiring Andy Payne into the Digital Cambridge > Research Lab. One of his little projects was a thing that > generated random but correct C programs and fed them to > different compilers or compilers with different switches to > see if they crashed or generated incorrect results. > Overnight, his tester filed 300 or so bug reports against the > Digital C compiler. This was met with substantial pushback, > but it was a mostly an issue that many of the reports traced > to the same underlying bugs. > > Bill McKeemon expanded the technique and published > "Differential Testing of Software" > https://www.cs.swarthmore.edu/~bylvisa1/cs97/f13/Papers/Differen= tialTestingForSoftware.pdf > > > In the mid-late 1980s Bill Mckeeman worked with DEC's compiler product > teams to introduce fuzz testing into our testing process. As with the > C compiler work at DEC Cambridge, fuzz testing for other compilers > (Fortran, PL/I) also found large numbers of bugs. > > The pushback from the compiler folks was mainly a matter of > priorities. Fuzz testing is very adept at finding edge conditions, > but most failing fuzz tests have syntax that no human programmer would > ever write. As a compiler engineer you have limited time to devote to > bug testing. Do you spend that time addressing real customer issues > that have been reported or do you spend it fixing problems with code > that no human being would ever write? To take an example that really > happened, a fuzz test consisting of 100 nested parentheses caused an > overflow in a parser table (it could only handle 50 nested parens). > Is that worth fixing? > > As you pointed out, fuzz test failures tend to occur in clusters and > many of the failures eventually are traced to the same underlying > bug. Which leads to the counter-argument to the pushback. The fuzz > tests are finding real underlying bugs. Why not fix them before a > customer runs into them? That very thing did happen several times. A > customer-reported bug was fixed and suddenly several of the fuzz test > problems that had been reported went away. Another consideration is > that, even back in the 1980s, humans weren't the only ones writing > programs. There were programs writing programs and they sometimes > produced bizarre (but syntactically correct) code. > > -Paul W. --------------1E2455C5EED03ABE5CCD444D Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

I like this anecdote because it points out the difference between being able to handle and process bizarre conditions, as if they were something that should work, which is maybe not that helpful, vs. detecting them and doing something reasonable, like failiing with a "limit exceeded" message. A silent, insidious failure down the line because a limit was exceeded is never good. If "fuzz testing" helps exercise limits and identifies places where software hasn't realized it has exceeded its limits, has run off the end of a table, etc., that seems like a good thing to me.

On 05/21/2024 09:59 AM, Paul Winalski wrote:
On Tue, May 21, 2024 at 12:09=E2=80=AFAM Serissa &l= t;stewart@serissa.com> wrote:
Well this is obviously a hot button topic.=C2=A0 AFAIK I was nearby when fuzz-testing for software was invented. I was the main advocate for hiring Andy Payne into the Digital Cambridge Research Lab.=C2=A0 One of his little projects was a thing that generated random but correct C programs and fed them to different compilers or compilers with different switches to see if they crashed or generated incorrect results.=C2=A0 Overnight, his test= er filed 300 or so bug reports against the Digital C compiler.=C2=A0 This was met with substantial pushback= , but it was a mostly an issue that many of the reports traced to the same underlying bugs.

Bill McKeemon expanded the technique and published "Differential Testing of Software"=C2=A0https://www.cs.swarthmore.edu/~byl= visa1/cs97/f13/Papers/DifferentialTestingForSoftware.pdf
=C2=A0
In the mid-late 1980s Bill Mckeeman worked with DEC's compiler product teams to introduce fuzz testing into our testing process.=C2=A0 As with the C compiler work at DEC Cambridge, fuzz testing for other compilers (Fortran, PL/I) also found large numbers of bugs.

The pushback from the compiler folks was mainly a matter of priorities.=C2=A0 Fuzz testing is very adept at finding edg= e conditions, but most failing fuzz tests have syntax that no human programmer would ever write.=C2=A0 As a compiler enginee= r you have limited time to devote to bug testing.=C2=A0 Do you spend that time addressing real customer issues that have been reported or do you spend it fixing problems with code that no human being would ever write?=C2=A0 To take an example that really happened, a fuzz test consisting of 100 nested parentheses caused an overflow in a parser table (it could only handle 50 nested parens).=C2=A0 Is that worth fixing?

As you pointed out, fuzz test failures tend to occur in clusters and many of the failures eventually are traced to the same underlying bug.=C2=A0 Which leads to the counter-argument to the pushback.=C2=A0 The fuzz tests are finding real underlying bugs.=C2=A0 Why not fix them before a customer runs into them?=C2=A0 That very thing did happen seve= ral times.=C2=A0 A customer-reported bug was fixed and suddenly several of the fuzz test problems that had been reported went away.=C2=A0 Another consideration is that, even back in t= he 1980s, humans weren't the only ones writing programs.=C2=A0 Th= ere were programs writing programs and they sometimes produced bizarre (but syntactically correct) code.

-Paul W.

--------------1E2455C5EED03ABE5CCD444D--