From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=5.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.4 Received: from minnie.tuhs.org (minnie.tuhs.org [50.116.15.146]) by inbox.vuxu.org (Postfix) with ESMTP id F29E420C3A for ; Tue, 21 May 2024 19:00:07 +0200 (CEST) Received: from minnie.tuhs.org (localhost [IPv6:::1]) by minnie.tuhs.org (Postfix) with ESMTP id 040FB43B71; Wed, 22 May 2024 03:00:00 +1000 (AEST) Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by minnie.tuhs.org (Postfix) with ESMTPS id 2FBF343B5E for ; Wed, 22 May 2024 02:59:51 +1000 (AEST) Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-5bdbe2de25fso586447a12.3 for ; Tue, 21 May 2024 09:59:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716310790; x=1716915590; darn=tuhs.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=NzvQmdKAc2R22aRW9mxzLCvzcTZV7zXYmBakTaYgt1Q=; b=coIciXedXxIqXp+nU88Jzq0WE98bnbMusqSrAsGdaFub9MjRl723wIKRYX0HSxFs0s 0RJ9LiF+cYVd6tOT3sIco+z4it7/XsItFTw3DvCpG99h4zUQrNSZ1AQ1iKjbFnwXZ5gG zKdeascK0K5wtMSRSfGDvf32dxQ6V4YxAWklBPTUbFIqrI6SwVJ3cJl2+6FoBUlUA0hQ hMS7PlwWe279otcATUAgaXf+3Qt2lEoLX4zyeZnkr/fEZUOZXdoPAo029N8uSG01YweF HWC04G2WkkXwLF+7M7JTbT+sKPr4gsQv8dWHFfUuR01rOfBkwm2MZuTi47h9uGvKqvPl togA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716310790; x=1716915590; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NzvQmdKAc2R22aRW9mxzLCvzcTZV7zXYmBakTaYgt1Q=; b=RYAbqaMCAijfZ2nn/ih2OQUMBNw4puh+A8McKJO4qmQKlFx8mtdShCvLnjBrhHnEGj I12YUsGxrRu+9oTxEfhmsk7jeyig4cKpcaOzvAbnAOGbDnv78m7F48/QV+hzWGo4u7uJ oJu5YjUUq5phOnqLr6qGaNHmx7RKWgM3jrc/j/+TjZfJ7fAv+ZIClLf3R71krAZyTi85 ukJIvw3iLP+mQ+76gqAKlJ0wZ+ORRsuTwhL2bcBU7qkxKKUeEHSH7Zt+a/EJy9NAKz/F H+MbGEUhFpTHy5TWLlA3yPd69IgubLRQoc9B+sBmWBNmF9LfgCTJP96Cr0Z/WwP7V6MJ 5VIg== X-Gm-Message-State: AOJu0YxLazwP85zGlNusoGBZ8Lexjq3QN0tJEpdMWwLm8Pl3ZdC93Oj8 DXTVpT9nWxmDme9Q9VC7X7mTuBLamJ+BNOYncpGgYNqNZTphoo313XJXBbNz97gj2XUwRYYyCGZ m6I1CGIpGVa3mBCn4WKWYcCpx3tuXPg== X-Google-Smtp-Source: AGHT+IEZD4HLJ6em2iUXGQ0vN8SMOv0Su4CRHJbOvcCP7UzarCVd+iGEmrx7hE0IBbDKRfsDy761nWuXF1k9XZgwYHg= X-Received: by 2002:a17:90a:6542:b0:2b3:be55:bf6f with SMTP id 98e67ed59e1d1-2b6cc76bdf6mr29799021a91.22.1716310790088; Tue, 21 May 2024 09:59:50 -0700 (PDT) MIME-Version: 1.0 References: <51CC9A0D-122C-4A3D-8BAF-C249489FB817@serissa.com> In-Reply-To: From: Paul Winalski Date: Tue, 21 May 2024 12:59:38 -0400 Message-ID: To: TUHS main list Content-Type: multipart/alternative; boundary="000000000000f7b2800618f9bee3" Message-ID-Hash: NWHGE5K2S2DJRUKWLNLEOPYSFVTHBNPX X-Message-ID-Hash: NWHGE5K2S2DJRUKWLNLEOPYSFVTHBNPX X-MailFrom: paul.winalski@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.6b1 Precedence: list Subject: [TUHS] Re: A fuzzy awk. List-Id: The Unix Heritage Society mailing list Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --000000000000f7b2800618f9bee3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, May 21, 2024 at 12:09=E2=80=AFAM Serissa wrot= e: > Well this is obviously a hot button topic. AFAIK I was nearby when >> fuzz-testing for software was invented. I was the main advocate for hiri= ng >> Andy Payne into the Digital Cambridge Research Lab. One of his little >> projects was a thing that generated random but correct C programs and fe= d >> them to different compilers or compilers with different switches to see = if >> they crashed or generated incorrect results. Overnight, his tester file= d >> 300 or so bug reports against the Digital C compiler. This was met with >> substantial pushback, but it was a mostly an issue that many of the repo= rts >> traced to the same underlying bugs. >> >> Bill McKeemon expanded the technique and published "Differential Testing >> of Software" >> https://www.cs.swarthmore.edu/~bylvisa1/cs97/f13/Papers/DifferentialTest= ingForSoftware.pdf >> > In the mid-late 1980s Bill Mckeeman worked with DEC's compiler product teams to introduce fuzz testing into our testing process. As with the C compiler work at DEC Cambridge, fuzz testing for other compilers (Fortran, PL/I) also found large numbers of bugs. The pushback from the compiler folks was mainly a matter of priorities. Fuzz testing is very adept at finding edge conditions, but most failing fuzz tests have syntax that no human programmer would ever write. As a compiler engineer you have limited time to devote to bug testing. Do you spend that time addressing real customer issues that have been reported or do you spend it fixing problems with code that no human being would ever write? To take an example that really happened, a fuzz test consisting of 100 nested parentheses caused an overflow in a parser table (it could only handle 50 nested parens). Is that worth fixing? As you pointed out, fuzz test failures tend to occur in clusters and many of the failures eventually are traced to the same underlying bug. Which leads to the counter-argument to the pushback. The fuzz tests are finding real underlying bugs. Why not fix them before a customer runs into them? That very thing did happen several times. A customer-reported bug was fixed and suddenly several of the fuzz test problems that had been reported went away. Another consideration is that, even back in the 1980s, humans weren't the only ones writing programs. There were programs writing programs and they sometimes produced bizarre (but syntactically correct) code. -Paul W. --000000000000f7b2800618f9bee3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, May 21, 2024 at 12:09=E2=80=AFAM Serissa <stewart@serissa.com= > wrote:
Well this is obviously a hot button topic.= =C2=A0 AFAIK I was nearby when fuzz-testing for software was invented. I wa= s the main advocate for hiring Andy Payne into the Digital Cambridge Resear= ch Lab.=C2=A0 One of his little projects was a thing that generated random = but correct C programs and fed them to different compilers or compilers wit= h different switches to see if they crashed or generated incorrect results.= =C2=A0 Overnight, his tester filed 300 or so bug reports against the Digita= l C compiler.=C2=A0 This was met with substantial pushback, but it was a mo= stly an issue that many of the reports traced to the same underlying bugs.<= br>

Bill McKeemon expanded the technique and published "= Differential Testing of Software"=C2=A0https://www.cs.swarthmore.edu/~bylvisa1/cs97/f13/Papers/Diff= erentialTestingForSoftware.pdf
=C2=A0
In the mid-late 1980s Bill Mckeeman worked wit= h DEC's compiler product teams to introduce fuzz testing into our testi= ng process.=C2=A0 As with the C compiler work at DEC Cambridge, fuzz testin= g for other compilers (Fortran, PL/I) also found large numbers of bugs.

The pushback from the compiler folks was mainly a mat= ter of priorities.=C2=A0 Fuzz testing is very adept at finding edge conditi= ons, but most failing fuzz tests have syntax that no human programmer would= ever write.=C2=A0 As a compiler engineer you have limited time to devote t= o bug testing.=C2=A0 Do you spend that time addressing real customer issues= that have been reported or do you spend it fixing problems with code that = no human being would ever write?=C2=A0 To take an example that really happe= ned, a fuzz test consisting of 100 nested parentheses caused an overflow in= a parser table (it could only handle 50 nested parens).=C2=A0 Is that wort= h fixing?

As you pointed out, fuzz test failur= es tend to occur in clusters and many of the failures eventually are traced= to the same underlying bug.=C2=A0 Which leads to the counter-argument to t= he pushback.=C2=A0 The fuzz tests are finding real underlying bugs.=C2=A0 W= hy not fix them before a customer runs into them?=C2=A0 That very thing did= happen several times.=C2=A0 A customer-reported bug was fixed and suddenly= several of the fuzz test problems that had been reported went away.=C2=A0 = Another consideration is that, even back in the 1980s, humans weren't t= he only ones writing programs.=C2=A0 There were programs writing programs a= nd they sometimes produced bizarre (but syntactically correct) code.

-Paul W.
--000000000000f7b2800618f9bee3--