Re: [TUHS] DEC Compilers (was: Re: SDB debugger

Re: [TUHS] DEC Compilers (was: Re: SDB debugger

[Doug McIlroy]

> random small C programs with computable expected outputs

"computable" is subtle here. The only way to compute the
outputs was to run the program. McKeeman's trick was to
sic several completely unrelated compilers on the program
and let them vote on the answer.

Compile time was measured. My favorite "bug" was the
mmany minutes it took to compile a constant expression
that involved shifting a constant INT_MAX bits by
performing that many 1-bit shifts.

Doug

Re: [TUHS] DEC Compilers (was: Re: SDB debugger

[Tim Rylance]

[-- Attachment #1: Type: text/plain, Size: 1570 bytes --]


>>> Compile time was measured. My favorite "bug" was the
>>> many minutes it took to compile a constant expression
>>> that involved shifting a constant INT_MAX bits by
>>> performing that many 1-bit shifts.
>> 
>> I don't know if this anecdote is an urban legend or if it really
>> happened.  I was told [a similar] story when I was interning as an operator
>> at my alma mater, which was an IBM System/360 shop.
> 
> I heard it not from the grapevine, but from McKeeman himself.

It’s mentioned in the paper (https://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf <https://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf>) on page 105, table 1

	Results of Testing C Compilers
	Source Code	Resulting Problem
	1>>INT_MAX	Twenty-minute compile time

but not explained.

My favourite is

	int(…(x)…)		enough nested parentheses to kill the compiler
				Spurious diagnostic (10 parentheses)
				Compiler crash (100 parentheses)
				Server crash (10,000 parentheses)

explained on page 104:

… the server crash occurred when the tested compiler got a stack overflow on a heavily loaded machine with a very large memory. The operating system attempted to dump a gigabyte of compiler stack, which caused all the other active users to thrash, and many of them also dumped for lack of memory. The many disk drives on the server began a dance of the lights that sopped up the remaining free resources, causing the operators to boot the server to recover. Excellent testing can make you unpopular with almost everyone. 


[-- Attachment #2: Type: text/html, Size: 2960 bytes --]

Re: [TUHS] DEC Compilers (was: Re: SDB debugger

[Doug McIlroy]

>> Compile time was measured. My favorite "bug" was the
>> many minutes it took to compile a constant expression
>> that involved shifting a constant INT_MAX bits by
>> performing that many 1-bit shifts.
>
> I don't know if this anecdote is an urban legend or if it really
> happened.  I was told [a similar] story when I was interning as an operator
> at my alma mater, which was an IBM System/360 shop.

I heard it not from the grapevine, but from McKeeman himself.

Doug

Re: [TUHS] DEC Compilers (was: Re: SDB debugger

[Dan Cross]

[-- Attachment #1: Type: text/plain, Size: 1902 bytes --]

On Tue, May 5, 2020 at 1:37 PM Paul Winalski <paul.winalski@gmail.com>
wrote:

> On 5/4/20, Win Treese <treese@acm.org> wrote:
> >
> > Andy Payne, a recent hire at the lab, had been an intern in DEC’s
> > semiconductor group, where he had worked on randomized testing for
> hardware
> > verification. With all the compilers available, he decided to hack up a
> > program to generate random small C programs with computable expected
> > outputs. His program then compiled the random code with each compiler and
> > tested the result. After finding a number of bugs this way, he got tired
> of
> > submitting the bug reports, and changed his program to write and submit
> the
> > bug reports automatically.
> >
> > This caused a little bit of consternation with some of the compiler
> teams at
> > first.
>
> I remember that very well.  IIRC it was called fuzz testing, and
> indeed it was controversial, for the reasons Bill McKeeman discusses
> in his paper. [1]  On the one hand, compiler developers said, "nobody
> would ever write something like that--we can't waste our time on these
> issues when there are real bugs waiting to be fixed."  On the other
> hand, some of the bugs that fuzz testing turned up provoked reactions
> such as, "OMG!  THAT caused the compiler to crash?"  I think the
> turning point was when fixing one of the fuzz testing bugs also fixed
> an obscure and hard-to-debug customer problem.  Intel's C and Fortran
> compiler team has also used random testing technology.
>

Ah, very cool. The same approach has come into favor again recently. I've
dealt personally with https://github.com/google/syzkaller, which is a
kernel fuzzer that generates random inputs to system calls and detects e.g.
panics. It's a neat approach.

        - Dan C.

> [1] https://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf
>
> -Paul W.
>

[-- Attachment #2: Type: text/html, Size: 2716 bytes --]

Re: [TUHS] DEC Compilers (was: Re: SDB debugger

[Henry Bent]

[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]

On Mon, 4 May 2020 at 20:33, Win Treese <treese@acm.org> wrote:

>
> > On May 3, 2020, at 4:26 PM, Clem Cole <clemc@ccc.com> wrote:
> >
> > Anyway back to compilers, Tru64 had a 'good enough' compiler based on
> the MIPS code base to get us all going, but GEM's primary target was VMS
> since one of the important features of GEM was the VAX->Alpha transpiler
> technology.   VMS was still heavily written in VAX Assembler at the time.
> Plus, It actually was a little hairy because GEM had a new C/C++
> front-end.   So TLE's high order bit was VMS for the Alphas.   GEM for
> Tru64 was about 18 months later.
>
> In the early days of Alpha, I was at DEC’s Cambridge Research Laboratory
> (directed then by Vic Vyssotsky, having retired from Bell Labs). The lab
> had various connections to Alpha projects, and we learned that there were
> (I think) 7 different C compilers running on the early port of Ultrix. That
> number, I think, did not include the port of gcc that DEC was funding
> outside the company.
>
> Andy Payne, a recent hire at the lab, had been an intern in DEC’s
> semiconductor group, where he had worked on randomized testing for hardware
> verification. With all the compilers available, he decided to hack up a
> program to generate random small C programs with computable expected
> outputs. His program then compiled the random code with each compiler and
> tested the result. After finding a number of bugs this way, he got tired of
> submitting the bug reports, and changed his program to write and submit the
> bug reports automatically.
>
> This caused a little bit of consternation with some of the compiler teams
> at first.
>
> Eventually, this led to some collaboration with the DEC languages and
> tools team, and Bill McKeeman published a paper that line of work in the
> Digital Technical Journal in 1998[1].
>
>  - Win
>
> [1] https://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf


Does this software still exist anywhere?  The link to the download is long
gone, archive.org did not preserve the download, and I had no success
finding the files on the web.

-Henry

[-- Attachment #2: Type: text/html, Size: 2685 bytes --]

Re: [TUHS] DEC Compilers (was: Re: SDB debugger

[Dr Iain Maoileoin]

> On 5 May 2020, at 18:36, Paul Winalski <paul.winalski@gmail.com> wrote:
> 
> On 5/4/20, Win Treese <treese@acm.org> wrote:
>> 
>> ….
>> This caused a little bit of consternation with some of the compiler teams at
>> first.
> 
> I remember that very well.  IIRC it was called fuzz testing, and
> ….
> compiler team has also used random testing technology.
> 
>> [1] https://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf
> 
> -Paul W.

Way back in the deep past - late 70s I think - The Computer Science Department at Strathclyde  Uni in Scotland had a contract to develop a test suite generator for the C compiler on the ICL perq computer.  I think the testing/development for that compiler was happening at Dalkeith in Scotland - but dont quote me.

Like the above we generated programs (e.g. mixing short, int, long signed and unsigned and doing all sort of ops on them).  The expected output was computed by the same C program running on a BSD unix vax and something else.  We had a few issues with the vax and the other system disagreeing on the arithmetic results, but generally we were confident the random C programs would reasonably test the system under test.  We did not get to see the results of the tests,  we developed the suite and handed it over to ICL.

Overall we were not impressed by the PERQ and on a trip to Rutherford Appleton Labs (RAL) one November there was a HUGE bonfire being prepared (for our Guy Falkes(sp) celebration).  The bonfire was generally comprised of the PERQ cardboard packing cases.  It just looked like they were planning to burn the PERQs themselves.  We agreed with the sentiment.

Re: [TUHS] DEC Compilers (was: Re: SDB debugger

[Paul Winalski]

On 5/4/20, Win Treese <treese@acm.org> wrote:
>
> Andy Payne, a recent hire at the lab, had been an intern in DEC’s
> semiconductor group, where he had worked on randomized testing for hardware
> verification. With all the compilers available, he decided to hack up a
> program to generate random small C programs with computable expected
> outputs. His program then compiled the random code with each compiler and
> tested the result. After finding a number of bugs this way, he got tired of
> submitting the bug reports, and changed his program to write and submit the
> bug reports automatically.
>
> This caused a little bit of consternation with some of the compiler teams at
> first.

I remember that very well.  IIRC it was called fuzz testing, and
indeed it was controversial, for the reasons Bill McKeeman discusses
in his paper. [1]  On the one hand, compiler developers said, "nobody
would ever write something like that--we can't waste our time on these
issues when there are real bugs waiting to be fixed."  On the other
hand, some of the bugs that fuzz testing turned up provoked reactions
such as, "OMG!  THAT caused the compiler to crash?"  I think the
turning point was when fixing one of the fuzz testing bugs also fixed
an obscure and hard-to-debug customer problem.  Intel's C and Fortran
compiler team has also used random testing technology.

> [1] https://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf

-Paul W.

Re: [TUHS] DEC Compilers (was: Re: SDB debugger

[Win Treese]

> On May 3, 2020, at 4:26 PM, Clem Cole <clemc@ccc.com> wrote:
> 
> Anyway back to compilers, Tru64 had a 'good enough' compiler based on the MIPS code base to get us all going, but GEM's primary target was VMS since one of the important features of GEM was the VAX->Alpha transpiler technology.   VMS was still heavily written in VAX Assembler at the time.  Plus, It actually was a little hairy because GEM had a new C/C++ front-end.   So TLE's high order bit was VMS for the Alphas.   GEM for Tru64 was about 18 months later. 

In the early days of Alpha, I was at DEC’s Cambridge Research Laboratory (directed then by Vic Vyssotsky, having retired from Bell Labs). The lab had various connections to Alpha projects, and we learned that there were (I think) 7 different C compilers running on the early port of Ultrix. That number, I think, did not include the port of gcc that DEC was funding outside the company.

Andy Payne, a recent hire at the lab, had been an intern in DEC’s semiconductor group, where he had worked on randomized testing for hardware verification. With all the compilers available, he decided to hack up a program to generate random small C programs with computable expected outputs. His program then compiled the random code with each compiler and tested the result. After finding a number of bugs this way, he got tired of submitting the bug reports, and changed his program to write and submit the bug reports automatically. 

This caused a little bit of consternation with some of the compiler teams at first.

Eventually, this led to some collaboration with the DEC languages and tools team, and Bill McKeeman published a paper that line of work in the Digital Technical Journal in 1998[1].

 - Win

[1] https://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf