Re: [9fans] how to avoid a memset() optimization

Re: [9fans] how to avoid a memset() optimization

[Russ Cox]

>   He doesn't need to know about volatile -- that's the whole point.
>   Once "experienced" programmer has set the buffer that way, the
>   less experienced just gets less chance to screw himself up

I'd still rather see it closer to the use that needs it.
If you're used to seeing secmemset then seeing
memset is a big red flag.  If you're used to seeing
memset with a volatile declaration, then you have
to seek out the volatile declaration, which might not
be anywhere near the call to memset.

This is a stupid argument anyway.  Volatile is a crock.
Here's another problem.  Suppose I'm worried that
leaking the processor flags at the end of my computation
might reveal something about the secret key.  I want to
clear them.  I could do something like:

	x = 0 & 0;

which in this particular case will clear the appropriate
flags.  But now the compiler comes along and optimizes
away the whole expression.  Where do I put my volatile
to make that stay?  Hmm?  The answer is not more crappy
qualifiers.  The answer is clearly dumber compilers.

Russ

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Russ Cox wrote:
> This is a stupid argument anyway.  Volatile is a crock.
> Here's another problem.  Suppose I'm worried that
> leaking the processor flags at the end of my computation
> might reveal something about the secret key.  I want to
> clear them.  I could do something like:
> 	x = 0 & 0;
> which in this particular case will clear the appropriate
> flags.  But now the compiler comes along and optimizes
> away the whole expression.  Where do I put my volatile
> to make that stay?  Hmm?  The answer is not more crappy
> qualifiers.  The answer is clearly dumber compilers.

Do you really want the compiler to always generate code
	MOV	R1,#0
	MOV	R2,#0
	AND	R2,R1
	MOV	R2,X(SP)
?  Almost any modern compiler will peform the AND at
compiler time (constant folding), and most customers want
that.  Anyway, I don't see why you need to AND in this
context.  Are you talking about the C and V bits?  If some
other process can inspect those for your process I'd think
it could inspect a whole lot more..

It is certainly true that we could really use better access
to the C bit from higher-level language source code.  This
shows up especially in the MP library.  But exploiting too
specific knowledge about the workings of a specific platform
can be dangerous, as we've seen historically.  If anybody
can devise a *nice* way to get at the C bit in MP coding,
please let me know so that it can be proposed for C0x.

Re: [9fans] how to avoid a memset() optimization

[Sam]

> Do you really want the compiler to always generate code
> 	MOV	R1,#0
> 	MOV	R2,#0
> 	AND	R2,R1
> 	MOV	R2,X(SP)

Yes, yes, and yes.  Optimizations which intelligently manage
register allocation are about as far as *I* appreciate.  Do
precisely what I say, and no less.  If I write poor code,
boo hoo for me.  Heavily optimizing compilers will *never*
make all coders equal and generally cause more headaches
than they're worth.

I do appreciate some warnings (ie if I forget to initialize
a variable to zero before I use it), but some of them are
just irritating.  Brantley looked over my shoulder a while
back and wondered why I was typing:

	if((m=fn())) {
		...
	}

saying, "what's with the extra parenthesis?"  "Oh, gcc complains
that I should surround assignment values used as a truth statement
and I've just incorporated it by habit to shut it up."

I hadn't even realized I was permitting gcc to define my style.
In essence, by permitting mad optimizations and then having to
do backflips just to keep the compiler from optimizing away
our real intentions, we're doing the same thing.  I hardly see
this as useful.

Cheers,

Sam

Re: [9fans] how to avoid a memset() optimization

[Scott Schwartz]

I, for one, like optimizing compilers.  Sure, sometimes you need fine-
grained control, but mostly I want faster code.  But like rob says, maybe
something other than C is what's called for.

Re: [9fans] how to avoid a memset() optimization

[Boyd Roberts]

I'd rather have slow and right over fast and broken.

Re: [9fans] how to avoid a memset() optimization

[plan9]

On Thu, Nov 14, 2002 at 04:26:08PM +0100, Boyd Roberts wrote:
> I'd rather have slow and right over fast and broken.

You and the other 5% of the world's consumers.

Re: [9fans] how to avoid a memset() optimization

[Sam]

Let's define "slow."  The last time I did a performance
test was on a des module I wrote.  The gcc compiled
-02 ran .2-.4 seconds faster (out of 3-4s).  Further
optlevels were not any better.

Anyone have any numbers showing that, "without optimizations,
this simply wouldn't be tolerable," in the face of well
written code?  In these instances does the compiler
generate crap code from the start, expecting an optimizer
to come clean it up later?

We wrote a C compiler for the language as of about
'73 for a coldfire target and it was a very telling
experience.  It was rather obvious that a simple
peephole optimizer would handle most of the inefficiencies
in the generated assy.  The code ran fast enough for
our purpose, however, so we didn't bother.  Indeed,
the human-time component is the most expensive
resource.  It's a shame 95% of the world doesn't
understand that.

Cheers,

Sam

On Thu, 14 Nov 2002 plan9@sigint.cs.purdue.edu wrote:

> On Thu, Nov 14, 2002 at 04:26:08PM +0100, Boyd Roberts wrote:
> > I'd rather have slow and right over fast and broken.
>
> You and the other 5% of the world's consumers.
>

Re: [9fans] how to avoid a memset() optimization

[Steve Kilbane]

> I'd rather have slow and right over fast and broken.

Others have different opinions. There are several issues here.

1. Some people want near-to-optimal code for the machine they've
chosen (slower code means faster machine needed, means higher price,
means fewer sales), and they want it soon (before their competitors
get all the market share). The economics of those markets mean they
need optimizing compilers. Vendors are happy to fill that need.

2. The language is being used for two different purposes: expressing
an abstract algorithm, and controlling the machine. These conflict.

3. There's a scary number of people out there who don't understand
the language they're using, and define it in terms of "Microsoft
gets it right," or "gcc does it properly", without realising they've
just been lucky. Their code can break from one compiler to another,
without having to optimize it. Optimizers just make it more obvious.

steve

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Steve's reponse was very nice.  Another way of putting it
is that the "right" behavior is not at all obvious once
you think hard enough about it and examine a wide enough
variety of platforms and application requirements.  One
of the tasks undertaken by the C standards committee was
to negotiate among the conflicting demands and to draw
the line somewhere.  Individuals would very likely draw
the line in other places, depending on their personal
prejudices and values.  If the existing standard
specification is causing significant real problems, there
is a mechanism in place to get it reconsidered and maybe
the official line could be redrawn.  The really important
thing is to understand where the official line *is* and
to observe it (on both sides, programmer and implementor);
that allows all parties to work together even if they all
still want to grumble about some aspect of the work..

Re: [9fans] how to avoid a memset() optimization

[Dan Cross]

I like Proebsting's law.  Compiler optimizations will get you a 2 fold
increase in speed once every 18 years.  That said, I wouldn't give up
the optimizations I have now, but I'm not married to them.

	- Dan C.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Dan Cross wrote:
> I like Proebsting's law.  Compiler optimizations will get you a 2
> fold increase in speed once every 18 years.

Do you mean, that optimization technology improves at that rate?
For RISC architectures it is hard to tell optimization apart from
simply doing a good job of allocating registers, pipelines, etc.

Re: [9fans] how to avoid a memset() optimization

[Dan Cross]

> > I like Proebsting's law.  Compiler optimizations will get you a 2
> > fold increase in speed once every 18 years.
>
> Do you mean, that optimization technology improves at that rate?

Yes.  He states it much more succinctly than I did:
http://research.microsoft.com/~toddpro/papers/law.htm

> For RISC architectures it is hard to tell optimization apart from
> simply doing a good job of allocating registers, pipelines, etc.

That's true, but eliminating whole function calls is pretty clearly
an optimization.

	- Dan C.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

> > Do you really want the compiler to always generate code
> >       MOV     R1,#0
> >       MOV     R2,#0
> >       AND     R2,R1
> >       MOV     R2,X(SP)
Sam wrote:
> Yes, yes, and yes.  Optimizations which intelligently manage
> register allocation are about as far as *I* appreciate.  Do
> precisely what I say, and no less.

I intentionally mimicked the PDP-11 in my example since it supports
nearly a direct mapping from C to instructions.  However, on many
modern machines there is no such direct mapping for many of (old)
C constructs, so trying to control code generation through use of
C source code is fruitless.  And once you accept that principle,
it is hard to tell what constitutes optimization versus simple
code generation.  If you want assembly language you know where to
find it...

Re: [9fans] how to avoid a memset() optimization

[Roman V. Shaposhnick]

On Thu, Nov 14, 2002 at 01:53:04AM -0500, Russ Cox wrote:
> >   He doesn't need to know about volatile -- that's the whole point.
> >   Once "experienced" programmer has set the buffer that way, the
> >   less experienced just gets less chance to screw himself up
>
> to make that stay?  Hmm?  The answer is not more crappy
> qualifiers.  The answer is clearly dumber compilers.

  Or may be the answer is
     #pragma get your hands off my data, your psycho

  ;)

Thanks,
Roman.

[9fans] presentation preparation

[Sam]

I'm working on a presentation and am
wondering what approach other 9'ers take
to preparing slides.  I'm thinking
a full screen page(1) would be sufficient
and I could use the extant doc prep
tools I'm becoming increasingly familiar
with.

Tips, tricks, and/or suggestions are welcomed.

Cheers,

Sam

Re: [9fans] presentation preparation

[Scott Schwartz]

| I could use the extant doc prep
| tools I'm becoming increasingly familiar
| with.

Not a Plan 9 specific answer, but...  I rather like latex with the
prosper package.  Ideally it is used to make pdf, and can take advantage
of acroread's features, but straight ps output should be usable.

Re: [9fans] presentation preparation

[andrey mirtchovski]

We were discussing latex presentations on this list some time ago (Russ
posted a fix to page to handle some mangled PostScript)...

I've done fine with presenting pdf or ps in a full-screen 'page' window on a
1024x768 laptop screen...

When talking about Plan 9 it is also useful to be able to hide the
presentation screen and show a few examples right there on the spot,
demo-style :)

I've sent Sam a sample presentation already, but didn't want to bother the
list with it. Another suggestion at the time was to use troff :)

andrey

On Sat, 15 Feb 2003, Scott Schwartz wrote:

> | I could use the extant doc prep
> | tools I'm becoming increasingly familiar
> | with.
>
> Not a Plan 9 specific answer, but...  I rather like latex with the
> prosper package.  Ideally it is used to make pdf, and can take advantage
> of acroread's features, but straight ps output should be usable.
>

Re: [9fans] presentation preparation

[Ralph Corderoy]

Hi andrey,

> Another suggestion at the time was to use troff :)

Bob Diertens' http://www.science.uva.nl/~bobd/useful/gpresent/ is useful
for producing PDF presentations using GNU troff.

Cheers,

--
Ralph Corderoy.      http://inputplus.co.uk/ralph/     http://troff.org/

[9fans] union directories

[Jack Johnson]

Ronald G. Minnich wrote:
 > truthfully, how many of you out there started out using disk/kfscmd
 > allow when you wanted to modify the kernel source?

I'll admit it, but I'm still a newbie so I have an extension of this
question.

Is there any way to, say, recompile a source tree with subfolders
without pre-creating those subfolders in the union directory?  Or,
what's the correct way to do something like:

mkdir 9
bind -bc 9 9
bind -ac /sys/src/9 9
cd 9
mk

(though that's not what I'm doing, I'm just needed an easy example)

-Jack

Re: [9fans] union directories

[Russ Cox]

there's no good way to do a recursive bind.
for the kernel you could piece it together with

t=/usr/you/9
bind -bc $t/pc /sys/src/9/pc
bind -bc $t/boot /sys/src/9/boot
cd /sys/src/9/pc
mk

Re: [9fans] union directories

[Adrian Tritschler]

Jack Johnson wrote:
> Ronald G. Minnich wrote:
>> truthfully, how many of you out there started out using disk/kfscmd
>> allow when you wanted to modify the kernel source?

I'll admit it.  How many knew any other way?  How many yet know?  Maybe a
little mention on the Wiki at
http://plan9.bell-labs.com/wiki/plan9/Compiling_kernels would be helpful.

Several months into my plan9 learning and I'm still getting my head around the
plan9 way of doing some things... most things...

> I'll admit it, but I'm still a newbie so I have an extension of this
> question.
> 
> Is there any way to, say, recompile a source tree with subfolders
> without pre-creating those subfolders in the union directory?  Or,
> what's the correct way to do something like:
> 
> mkdir 9
> bind -bc 9 9
> bind -ac /sys/src/9 9
> cd 9
> mk
> 
> (though that's not what I'm doing, I'm just needed an easy example)
> 
> -Jack

	Adrian

---------------------------------------------------------------
Adrian Tritschler    mailto:Adrian.Tritschler@its.monash.edu.au
Latitude 38°S, Longitude 145°E, Altitude 50m,      Shoe size 44
---------------------------------------------------------------

Re: [9fans] presentation preparation

[Richard Miller]

> I'm working on a presentation and am
> wondering what approach other 9'ers take
> to preparing slides.

I write slides in html [actually in a simpler syntax which is
translated to html by an awk script] and use charon to display
them.

-- Richard

Re: [9fans] presentation preparation

[Russ Cox]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

Attached are some presentations I've done using
troff and page, along with tools that make them
easier.

You can play with the variables at the end of tmac.ppt
(or the definitions themselves) to add color to various
parts.

I tend to shy away from color usually, though.
The auth talk is black and white, while the oscpu
presentation adds grey.

Russ

[-- Attachment #2: doc.tar.gz --]
[-- Type: application/octet-stream, Size: 12370 bytes --]

Re: [9fans] presentation preparation

[Fco.J.Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 168 bytes --]

An alternative is to use either the foils macro package or
a derivative. This is the one I use along with the respective mkfile
with view, print, ... targets

hth

[-- Attachment #2: slides.tgz --]
[-- Type: application/x-tar, Size: 7848 bytes --]

Re: [9fans] how to avoid a memset() optimization

[nigel]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

Or

        failed = 1;
        if ((res1 = lock(A) != 0)
	        if ((res2 = lock(B) != 0) {
		        if ((res3 = lock(C) != 0) {
				foo(res1, res2, res3);
				failed = 0;
			}
	if (res3 != 0)
        		release(C);
	if (res2 != 0)
        		release(B);
	if (res1 != 0)
        		release(A);

        return failed;

avoiding the goto.

Further, if release(nil) is implemented as a noop you can have

        failed = 1;
        if ((res1 = lock(A) != 0)
	        if ((res2 = lock(B) != 0) {
		        if ((res3 = lock(C) != 0) {
				foo(res1, res2, res3);
				failed = 0;
			}
	release(C);
	release(B);
	release(A);

        return failed;

[-- Attachment #2: Type: message/rfc822, Size: 2464 bytes --]

From: Ralph Corderoy <ralph@inputplus.co.uk>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] how to avoid a memset() optimization
Date: Mon, 6 Jan 2003 10:18:19 GMT
Message-ID: <3afa.3e16dd11.67d1a@blake.inputplus.co.uk>

Hi Roman,

> Of course, good old:
>      
>      if ((a = do_it()) == BadThingHappened)
>           return TellEmAboutIt;
> 
> works, but costs a little bit too much when clarity is needed.
> Especially when a "transaction" like pattern is needed ( e.g. I need
> to to have o1 = f1(); o2 = f2(); o3 = f3(); but if any of f* fails, I
> have to destroy what I've got so far ). 

A common way of doing this is using goto.

        failed = 1;

        if ((res1 = lock(A) == 0) {
            goto releaseA;
        }
        if ((res2 = lock(B) == 0) {
            goto releaseB;
        }
        if ((res3 = lock(C) == 0) {
            goto releaseC;
        }

        foo(res1, res2, res3);
        failed = 0;

    releaseC:
        release(C);
    releaseB:
        release(B);
    releaseA:
        release(A);

        return failed;

Cheers,

-- 
Ralph Corderoy.      http://inputplus.co.uk/ralph/     http://troff.org/

Re: [9fans] how to avoid a memset() optimization

[Geoff Collyer]

One can find examples of this technique in /sys/src/fs.
Here's some clean-up code from the implementation of wstat
at /sys/src/fs/port/9p1.c:1499,1504:

	if(p)
		putbuf(p);
	if(p1)
		putbuf(p1);
	if(f)
		qunlock(f);

Re: [9fans] how to avoid a memset() optimization

[presotto]

The exceptional style has its merits.  If C had a resonable multivalued
return mechanism like alef or limbo, I think it would have been a better
solution.

There's no reason other than personal preference to not use the same
model at user level, it's just a stack of long jmp variables.  However,
I haven't found the need.  Return values seem to work just as well
though they are much wordier.

Re: [9fans] how to avoid a memset() optimization

[Tomas]

On Tue, 19 Nov 2002 at 9:32am, presotto@plan9.bell-labs.com wrote:

> The exceptional style has its merits.  If C had a resonable
> multivalued return mechanism like alef or limbo, I think it
> would have been a better solution.
>
> There's no reason other than personal preference to not use the
> same model at user level, it's just a stack of long jmp
> variables.  However, I haven't found the need.  Return values
> seem to work just as well though they are much wordier.

I've been fantasizing about having some sort of atreturn()
functionality similar to atexit() to use in error handling (as
soon as a file is opened, do an atreturn() and specify that that
fd should be closed upon a "return -1;" but not otherwise, for
example).

Has anyone else been thinking in that direction? Does anyone have
references to mailing list/usenet discussions or papers about
such a feature?

/Tomas

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Tomas wrote:
> I've been fantasizing about having some sort of atreturn()
> functionality similar to atexit() to use in error handling (as
> soon as a file is opened, do an atreturn() and specify that that
> fd should be closed upon a "return -1;" but not otherwise, for
> example).
> Has anyone else been thinking in that direction? Does anyone have
> references to mailing list/usenet discussions or papers about
> such a feature?

Back in the Good Old Days (tm), we had what we called "completion
routines".  In an I/O service call one could optionally include a
pointer to a completion routine, which the OS would invoke after
the operation was complete.  (Of course the service call returned
immediately without waiting for the operation to complete.)

In C you could invent an API implemented as macros that would do
more or less what you indicated; there would have to be a label
near the end of the function.  It's not obvious that this buys
you very much, unlike general exception handling which spans
multiple function depth.

Re: [9fans] how to avoid a memset() optimization

[Fco.J.Ballesteros]

>   That's a very interesting remark, since I've always wondered why don't
>   you use waserror()/nexterror() in the rest of the Plan9 source tree
>   as a regular error handling mechanism.

But they still control the compilers for Plan 9 non-kernel code.
In fact, I'm using the same kind of stuff for error handling in
several programs.

[9fans] how to avoid a memset() optimization

[Joel Salomon]

Andrew Simmons wrote:
>http://groups.google.com/groups?threadm=oA47GsAf7M29EwY0%40robinton.demon.co.uk
this did not work for me, try:
http://groups.google.com/groups?hl=en&lr=lang_en&ie=UTF-8&threadm=3DD35AAB.AA37D5CA%40monitorbm.co.nz&rnum=1
(should be all on one line)

--Joel
______________________________________________________
Due to economic circumstances, the light at the end of
the tunnel has been turned off.

Re: [9fans] how to avoid a memset() optimization

[Andrew Simmons]

I was perturbed by Ron's remarks on structure layout in C++, and
posted a query on one of the C++ groups. I wasn't intending to draw
attention to it here, but the discussion has reached a stage that
might appeal to connoisseurs of arcana:

http://groups.google.com/groups?threadm=oA47GsAf7M29EwY0%40robinton.de
mon.co.uk

Re: [9fans] how to avoid a memset() optimization

[C H Forsyth]

>>#define ____cacheline_aligned __attribute__((__aligned__(SMP_CACHE_BYTES)))

oh dear.  how do they come up with this stuff?

i think it's a Martian invasion, actually.

Re: [9fans] how to avoid a memset() optimization

[Dennis Ritchie]

Gwyn remarked

 > C as you imagine it was never really in existence; even on
 > the PDP-11, certain access patterns (such as dereference
 >  of constants in the range 0160000-017777t) were specially
 > (kludge) excluded from optimization, so that device drivers
 > wouldn't be broken by non-1-to-1 translations from source
 > code to accesses.

This wasn't true for the PDP-11 compiler, which
(approximately) treated everything as volatile in the
current sense, though it could do some elementary
dead-code elimination.  The (early) BSD/VAX assembly
optimizer did have something like this, however, in that it
avoided certain instructions that wouldn't work in the I/O
space.

 > "volatile" addresses such issues as
 > well as we could with a simple portable mechanism.

The C90 and C99 semantics of volatile remain a bit flabby,
though it would take a lot of work to get a standard
that captured the behavior.  It's been tried.

The intent of volatile was to capture appropriate
behavior of memory-mapped registers and similar
things (like a clock in user space updated by the
OS.)  So, things like

	*p = 0;
	*p = 0;

should generate two stores if p is volatile *int.

One thing added in C90 was volatile's use in proper preservation
of values of automatics in the presence of
setjmp/longjmp. This was a non-issue for the PDP-11
because of details of its calling sequence, harder but
not impossible for the VAX because of its more complicated
(and slow) calling sequence, a real mess with the
advent of RISCs and optimization.  This is presumably
what hit Dave etc. with the setjmp-like waserror/nexterror
(the compiler didn't trigger on those words the
way they're supposed to on setjmp/longjmp).

	Dennis

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Dennis Ritchie wrote:
> This wasn't true for the PDP-11 compiler, which
> (approximately) treated everything as volatile in the
> current sense, though it could do some elementary
> dead-code elimination.  The (early) BSD/VAX assembly
> optimizer did have something like this, however, in that it
> avoided certain instructions that wouldn't work in the I/O
> space.

Thanks for the correction.  I was told about it by people
who worked on the PCC/QCC/RCC etc. family in general, and
from the description I thought they meant the PDP-11;
of course the VAX-11 had a very similar I/O interface.
Regardless, the PDP-11 even with its relatively direct
mapping of C into instructions still required careful
fiddling in the source code for some device drivers in
order to avoid read-modify-write cycles etc. -- stuff
that is clearly beyond the scope of what could be
controlled with a single keyword, much less what could
be gotten "right" in every case with no work for the
programmer.

 > The intent of volatile was to capture appropriate
 > behavior of memory-mapped registers and similar
 > things (like a clock in user space updated by the
 > OS.)  So, things like
 > 	*p = 0;
 > 	*p = 0;
 > should generate two stores if p is volatile *int.

Yes, the C standard requires that, with the correction
that it is the int that must be volatile-qualified,
not the pointer.  I.e., volatile int* if we're using C
abstract types.  It is still up to the implementation
to determine whether the store involves a read also
and how wide the access is (e.g., if int is 32 bits on
a 64-bit word bus, the store would necessitate fetch
of 64 bits, modification of 32 of them, and write-back
of 64 bits).  There doesn't seem to be any point in
trying to let the programmer specify such details,
since they're normally built into the hardware.  But
volatile as it is specified at least lets the programmer
control the *compiler* (code generator), which is
partial control and quite often good enough.

Re: [9fans] how to avoid a memset() optimization

[Boyd Roberts]

Dennis Ritchie wrote:

>... The (early) BSD/VAX assembly
>optimizer did have something like this, however, in that it
>avoided certain instructions that wouldn't work in the I/O
>space.
>
>
>
And on the VAX this would probably be a large list.  The 8th
Edition compiler/assembler would output extract field instructions
for some 16 bit operations which broke on UNIBUS device
registers (the tu11 tape driver iirc).

Re: [9fans] how to avoid a memset() optimization

[jmk]

On Thu Nov 14 13:32:18 EST 2002, tad@entrisphere.com wrote:
>
> someone mentioned something along the lines of
>
> void*
> secmemset(void *buf, int v, int len)
> {
> 	return memset((volatile void*)buf, v, len);
> }
>
> However, isn't GCC going to complain that you're passing
> a volatile to a function that isn't expecting a volatile?
>
> -Tad

Jarring chord. The door flies open and Cardinal Ximinez of Spain enters
flanked by two junior cardinals. Cardinal Biggles has goggles pushed over
his forehead. Cardinal Fang is just Cardinal Fang.

Ximinez:
	Nobody expects the Volatile Inquisition. Our chief weapon is surprise
	that your code doesn't work ... surprise and fear that you forgot an
	optimisation parameter... fear and surprise ... our two weapons are fear
	and surprise ... and ruthless efficiency instead of correctness. Our
	THREE weapons are fear, surprise and ruthless efficiency and an almost
	fanatical devotion to RMS ... Our FOUR ... no ... AMONGST our weaponry
	are such elements as fear, surprise ... I'll come in again.
	(exit and exeunt)

Re: [9fans] how to avoid a memset() optimization

[Steve Kilbane]

> 	Our FOUR ... no ... AMONGST our weaponry
> 	are such elements as fear, surprise ... I'll come in again.

So it's reentrant, too?

Re: [9fans] how to avoid a memset() optimization

[presotto]

[-- Attachment #1: Type: text/plain, Size: 107 bytes --]

nah, the inability to do a good job of something has
never stopped us before.  it's just lack of interest.

[-- Attachment #2: Type: message/rfc822, Size: 2113 bytes --]

From: Joel Salomon <salomo3@cooper.edu>
To: 9fans@cse.psu.edu
Subject: [9fans] how to avoid a memset() optimization
Date: Thu, 14 Nov 2002 13:11:17 -0500 (EST)
Message-ID: <Pine.SOL.3.96.1021114130542.22864A-100000@robin.cooper.edu>

> Anyone have any numbers showing that, "without optimizations,
> this simply wouldn't be tolerable," in the face of well
> written code?
I believe this is the case with the IA-64 architecture. I don't have any
experience writing compilers, but I would imagine that compile-time
concurrency analysis (necessary for *decent* performance on IA-64)
requires very clever optimizers.
Come to think of it, I don't believe that plan9's comilers have been
ported to IA-64 - is this the reason?

 --Joel
______________________________________________________
Due to economic circumstances, the light at the end of
the tunnel
has been
turned off.

[9fans] how to avoid a memset() optimization

[Joel Salomon]

> Anyone have any numbers showing that, "without optimizations,
> this simply wouldn't be tolerable," in the face of well
> written code?
I believe this is the case with the IA-64 architecture. I don't have any
experience writing compilers, but I would imagine that compile-time
concurrency analysis (necessary for *decent* performance on IA-64)
requires very clever optimizers.
Come to think of it, I don't believe that plan9's comilers have been
ported to IA-64 - is this the reason?

 --Joel
______________________________________________________
Due to economic circumstances, the light at the end of
the tunnel
has been
turned off.

Re: [9fans] how to avoid a memset() optimization

[William Josephson]

On Thu, Nov 14, 2002 at 01:11:17PM -0500, Joel Salomon wrote:
> > Anyone have any numbers showing that, "without optimizations,
> > this simply wouldn't be tolerable," in the face of well
> > written code?
> I believe this is the case with the IA-64 architecture. I don't have any
> experience writing compilers, but I would imagine that compile-time
> concurrency analysis (necessary for *decent* performance on IA-64)
> requires very clever optimizers.

I certainly haven't heard anything kind from
compiler writers about IA-64.

> Come to think of it, I don't believe that plan9's comilers have been
> ported to IA-64 - is this the reason?

I would guess that it has more to do with
the fact that no one actually uses IA64.
Porting the compilers is a lot of work if
you aren't going to use them :-)

Re: [9fans] how to avoid a memset() optimization

[rog]

> memset((volatile void*)p,n,0) would be similar if it were allowed.

that's odd.

i thought it had originally been suggested that:

void
f(void)
{
	volatile void *buf = malloc(n);

	memset(buf, n, 0);
}

was the "right" way to do it.
but now it appears that's not allowed...
does that mean that in fact there's *no* portable way
of doing it?!

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

rog@vitanuova.com wrote:
> i thought it had originally been suggested that:

The original original suggestion was to declare the buffer
object itself with volatile qualification.

Re: [9fans] how to avoid a memset() optimization

[Russ Cox]

> Do you really want the compiler to always generate code
> 	MOV	R1,#0
> 	MOV	R2,#0
> 	AND	R2,R1
> 	MOV	R2,X(SP)
> ?  Almost any modern compiler will peform the AND at
> compiler time (constant folding), and most customers want
> that.  Anyway, I don't see why you need to AND in this
> context.  Are you talking about the C and V bits?  If some
> other process can inspect those for your process I'd think
> it could inspect a whole lot more..

i _said_ it was a stupid argument.  my point was more
that volatile addresses memory optimizations but not
necessarily other things which might matter just as much.

Re: [9fans] how to avoid a memset() optimization

[presotto]

My probably incorrect 2 cents.  I think the basic problem is
bigger than volatile's power to solve.  There is no silver bullet.

My only experience with volatile was trying to make inferno emu code
run when compiled with non-plan9 compilers.  I never did figure out
everywhere it was needed.  Our waserror()/nexterror() stuff completely
blew away optimizing compilers since they are essentially longjumps.
Therefore, the compilers didn't know the active scope of a variable
so that registerization (and other opts) broke the code.  The appropriate
application of volatile fixed the problem though appropriate seemed
to be different for each compiler.  The general rule of thumb was that
any variable used by the waserror block should be volatile.  This seemed
to work in most instances but not all.  We had afternoons of sitting around
the Unix room trying to define exactly what it was that volatile
did in different compilers.  I never felt safe about the emu code
because of it.

My three main conclusions from this were:
1) waserror/nexterror is definitely evil unless understood by the
 compiler and perhaps even then.  It's a step outside the language
 definition and therefore a dangerous step to take.  We were
 comfortable with it in Plan 9 because we controlled the compilers
 but it became tortuous when someone else controlled them.

2) Volatile's meaning needs to grow to encompass the optimization
 du jour (don't registerize, don't optimize away memory accesses,
 don't optimize away condition code changes, don't inline code
 that contains it, ...) or we need more constructs.  It's hard to
 not screw up when using a construct whose meaning requires an in
 depth knowledge of what the compiler does.  To a certain extent, knowing
 the compiler's properties is a prerequisite to working in a kernel
 but there's a limit to what you have to understand.

 I'ld be happy if volatile just meant what I think it was originally
 intended for: don't optimize away or reorder any memory references.
 However, that in itself has myriad side effects.
 People will start to use it to avoid loop unrollings etc not envisioned
 by the compiler writer.

3) I'm too stupid to understand what C has become.  Perhaps I should go back
 to assembler.  Oops, I'm way to stupid to understand what ken's
 assembler does, I should go back to 1's and 0's.

The idea that compiler optimization is a knob that you turn till some
assumption you made becomes incorrect is scarey to me.  Very few
people understand the languages they use well enough to know when
they're treading on dangerous ground.  Even fewer testing environments
are complete enough to notice when something really has broken such
as the inadvertant creation of covert channels that got this started.
Luckily incorrect behavior in most programs doesn't really matter
because what most programs do is pretty ill defined.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

presotto@plan9.bell-labs.com wrote:
> The idea that compiler optimization is a knob that you turn
> till some assumption you made becomes incorrect is scarey to me.

A major purpose of the language standard is to define the
"treaty point": what a careful programmer has a right to expect
from the implementation and what latitude the implementation is
allowed.  The problem with assumptions is that there is no
standard for them.

Re: [9fans] how to avoid a memset() optimization

[William Josephson]

On Fri, Nov 15, 2002 at 10:50:38AM +0000, Douglas A. Gwyn wrote:
> presotto@plan9.bell-labs.com wrote:
> > The idea that compiler optimization is a knob that you turn
> > till some assumption you made becomes incorrect is scarey to me.
>
> A major purpose of the language standard is to define the
> "treaty point": what a careful programmer has a right to expect
> from the implementation and what latitude the implementation is
> allowed.  The problem with assumptions is that there is no
> standard for them.

While this is a fine sentiment, it is, in practice, not so easily
applied by the programmer.  In general, it is easier for run of
the mill applications and much harder for operating systems.  Can
you name any major operating system written in C that actually
compiles reliably with more than one compiler?  How many even work
with different major releases of the compiler?  Some of this is
probably due to sloppiness, but I don't think it is reasonable to
claim that it is only due to sloppiness.  The fact of the matter
seems to be that in the real world the standard, although important,
isn't as useful as you're claiming.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

William Josephson wrote:
> Can you name any major operating system written in C that actually
> compiles reliably with more than one compiler?

Unix used to; the real-time embedded OS I'm using (RTXC) does.
All my embedded systems code does.  It's not hard, really, if
you code to the standard.  There are certain operations that
can be difficult on some architectures, such as the VAX and
PDP-11 access pattern requirements that we've discussed, but
(some) newer architectures are more robustly designed.

Re: [9fans] how to avoid a memset() optimization

[Ronald G. Minnich]

On Mon, 18 Nov 2002, Douglas A. Gwyn wrote:

> William Josephson wrote:
> > Can you name any major operating system written in C that actually
> > compiles reliably with more than one compiler?
>
> Unix used to; the real-time embedded OS I'm using (RTXC) does.

wow, missed that question and the answer.

Things have sure gone downhill, eh? Our OS source code, written in C, is
less portable now than 14 years ago. Oh, well, what's a little of this
among friends:

#define ____cacheline_aligned __attribute__((__aligned__(SMP_CACHE_BYTES)))

doesn't every C compiler support that?

I, sir, am a curmudgeon. I have it on the best authority.

ron

Re: [9fans] how to avoid a memset() optimization

[Roman V. Shaposhnick]

On Thu, Nov 14, 2002 at 11:47:11AM -0500, presotto@plan9.bell-labs.com wrote:
> Our waserror()/nexterror() stuff completely
> blew away optimizing compilers since they are essentially longjumps.
........
> My three main conclusions from this were:
> 1) waserror/nexterror is definitely evil unless understood by the
>  compiler and perhaps even then.  It's a step outside the language
>  definition and therefore a dangerous step to take.  We were
>  comfortable with it in Plan 9 because we controlled the compilers
>  but it became tortuous when someone else controlled them.

  That's a very interesting remark, since I've always wondered why don't
  you use waserror()/nexterror() in the rest of the Plan9 source tree
  as a regular error handling mechanism.

  Personally I'm still struggling with developing an error handling
  policy that I'd feel comfortable with. Of course, good old:

       if ((a = do_it()) == BadThingHappened)
            return TellEmAboutIt;

  works, but costs a little bit too much when clarity is needed. Especially
  when a "transaction" like pattern is needed ( e.g. I need to
  to have o1 = f1(); o2 = f2(); o3 = f3(); but if any of f* fails,
  I have to destroy what I've got so far ).

  C++ style exceptions are nice, but easily abused, and I haven't seen
  any way of emulating then in pure C.

  So, what your experience has been with error handling ( granted, you
  must've had plenty ;-) ) and what would you consider the most
  comfortable one ?

Thanks,
Roman.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Roman V. Shaposhnick wrote:
>   C++ style exceptions are nice, but easily abused,
 > and I haven't seen any way of emulating then in pure C.

Nearly any programming facility can be abused.
Of course C++ exceptions cannot be exactly emulated
in C, because there is no possible way to tie the
handling to the type as in C++.  However, similar
facilities can be built on top of setjmp/longjmp,
and several people have done so.  Send me e-mail at
my work address and I'll be happy to send you my
implementation which I call the "Ex" package.

Re: [9fans] how to avoid a memset() optimization

[Roman V. Shaposhnick]

[ DISCLAIMER: sorry for wide distribution, but I had no luck contacting
  Douglas at his e-mail ].

On Wed, Nov 20, 2002 at 09:47:53AM +0000, Douglas A. Gwyn wrote:
> Roman V. Shaposhnick wrote:
> >   C++ style exceptions are nice, but easily abused,
>  > and I haven't seen any way of emulating then in pure C.
>
> Nearly any programming facility can be abused.

  Oh, that's for sure. I guess I just have a "sausage
  problem" with C++ ( you, know, people who make 'em,
  can't eat 'em ) since I've been involved in C++ compiler
  development for quite some time. Since then, I'm
  a pure C diehard but I miss very basic functionality
  of what was known as "C with classes" from time to time.

> Of course C++ exceptions cannot be exactly emulated
> in C, because there is no possible way to tie the
> handling to the type as in C++.  However, similar
> facilities can be built on top of setjmp/longjmp,
> and several people have done so.  Send me e-mail at
> my work address and I'll be happy to send you my
> implementation which I call the "Ex" package.

  That'll be wonderful. Thank you.

Thanks,
Roman.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

"Roman V. Shaposhnick" wrote:
> [ DISCLAIMER: sorry for wide distribution, but I had no luck
> contacting Douglas at his e-mail ].

I got the e-mail, but the package I sent in response got
queued up due to inability to connect directly to your
host.  If it doesn't arrive within a couple of days
send me another note.

Re: [9fans] how to avoid a memset() optimization

[Ralph Corderoy]

Hi Roman,

> Of course, good old:
>      
>      if ((a = do_it()) == BadThingHappened)
>           return TellEmAboutIt;
> 
> works, but costs a little bit too much when clarity is needed.
> Especially when a "transaction" like pattern is needed ( e.g. I need
> to to have o1 = f1(); o2 = f2(); o3 = f3(); but if any of f* fails, I
> have to destroy what I've got so far ). 

A common way of doing this is using goto.

        failed = 1;

        if ((res1 = lock(A) == 0) {
            goto releaseA;
        }
        if ((res2 = lock(B) == 0) {
            goto releaseB;
        }
        if ((res3 = lock(C) == 0) {
            goto releaseC;
        }

        foo(res1, res2, res3);
        failed = 0;

    releaseC:
        release(C);
    releaseB:
        release(B);
    releaseA:
        release(A);

        return failed;

Cheers,

-- 
Ralph Corderoy.      http://inputplus.co.uk/ralph/     http://troff.org/

Re: [9fans] how to avoid a memset() optimization

[Sam]

or:

{
	int m=0;

	if(++m && (res1 = lock(A)))
	if(++m && (res2 = lock(B)))
	if(++m && (res3 = lock(C))) {
		foo();
		return success;
	}
	switch(m) {
	case 3:	 release(B);
	case 2:  release(A);
	default: return failure;
	}
}

you just need a marker to see how far you got. (I'm not even sure
what the context of this message is, but every time i see the goto
solution to this I have to say something). ;)

Cheers,

Sam

>
>         failed = 1;
>
>         if ((res1 = lock(A) == 0) {
>             goto releaseA;
>         }
>         if ((res2 = lock(B) == 0) {
>             goto releaseB;
>         }
>         if ((res3 = lock(C) == 0) {
>             goto releaseC;
>         }
>
>         foo(res1, res2, res3);
>         failed = 0;
>
>     releaseC:
>         release(C);
>     releaseB:
>         release(B);
>     releaseA:
>         release(A);
>
>         return failed;
>
> Cheers,
>
>

Re: [9fans] how to avoid a memset() optimization

[Russ Cox]

> you just need a marker to see how far you got. (I'm not even sure
> what the context of this message is, but every time i see the goto
> solution to this I have to say something). ;)

the goto seems much clearer to me.
also easier to edit.  what happens when
you have to lock A½?  you have to rewrite
all your case statements!

g% grep goto /sys/games/lib/fortunes |sed 1q
If you want to go somewhere, goto is the best way to get there. K Thompson
g% 

also you can't always write the code in your if-style,
but you can always use a goto.  for example, you can't
rewrite this without goto (or at least, if you do, it ends
up a lot less clear or a lot more duplicated):

static int
p9skclient(Conv *c)
{
	char *user;
	char cchal[CHALLEN];
	uchar secret[8];
	char buf[MAXAUTH];
	int speakfor, ret;
	Attr *a;
	Authenticator au;
	Key *k;
	Ticket t;
	Ticketreq tr;

	ret = -1;
	a = nil;
	k = nil;

	/* p9sk1: send client challenge */
	if(c->proto == &p9sk1){
		c->state = "write challenge";
		memrandom(cchal, CHALLEN);
		if(convwrite(c, cchal, CHALLEN) < 0)
			goto out;
	}

	/* read ticket request */
	c->state = "read tickreq";
	if(convread(c, buf, TICKREQLEN) < 0)
		goto out;
	convM2TR(buf, &tr);

	/* p9sk2: use server challenge as client challenge */
	if(c->proto == &p9sk2)
		memmove(cchal, tr.chal, CHALLEN);

	/*
	 * find a key.
	 *
	 * if the user is the factotum owner, any key will do.
	 * if not, then if we have a speakfor key,
	 * we will only vouch for the user's local identity.
	 *
	 * this logic is duplicated in p9any.c
	 */
	user = strfindattr(c->attr, "user");
	a = delattr(copyattr(c->attr), "role");
	a = addattr(a, "proto=p9sk1");

	if(strcmp(c->sysuser, owner) == 0){
		speakfor = 0;
		a = addattr(a, "proto=p9sk1 user? dom=%q", tr.authdom);
	}else if(user==nil || strcmp(c->sysuser, user)==0){
		speakfor = 1;
		a = delattr(a, "user");
		a = addattr(a, "proto=p9sk1 user? dom=%q role=speakfor", tr.authdom);
	}else{
		werrstr("will not authenticate for %q as %q", c->sysuser, user);
		goto out;
	}

	for(;;){
		c->state = "find key";
		k = keyfetch(c, "%A", a);
		if(k == nil)
			goto out;
		
		/* relay ticket request to auth server, get tickets */
		strcpy(tr.hostid, strfindattr(k->attr, "user"));
		if(speakfor)
			strcpy(tr.uid, c->sysuser);
		else
			strcpy(tr.uid, tr.hostid);

		c->state = "get tickets";
		if(gettickets(&tr, buf, k) < 0)
			goto out;

		convM2T(buf, &t, k->priv);
		if(t.num == AuthTc)
			break;

		/* we don't agree with the auth server about the key; try again */
		c->state = "replace key";
		if((k = keyreplace(c, k, "key mismatch with auth server")) == nil){
			werrstr("key mismatch with auth server");
			goto out;
		}
	}

	/* send second ticket and authenticator to server */
	c->state = "write ticket+auth";
	memmove(buf, buf+TICKETLEN, TICKETLEN);
	au.num = AuthAc;
	memmove(au.chal, tr.chal, CHALLEN);
	au.id = 0;
	convA2M(&au, buf+TICKETLEN, t.key);
	if(convwrite(c, buf, TICKETLEN+AUTHENTLEN) < 0)
		goto out;

	/* read authenticator from server */
	c->state = "read auth";
	if(convread(c, buf, AUTHENTLEN) < 0)
		goto out;
	convM2A(buf, &au, t.key);
	if(au.num != AuthAs || memcmp(au.chal, cchal, CHALLEN) != 0 || au.id != 0){
		werrstr("server lies through his teeth");
		goto out;
	}

	/* success */
	c->attr = addcap(c->attr, c->sysuser, &t);
	des56to64((uchar*)t.key, secret);
	c->attr = addattr(c->attr, "secret=%.8H", secret);
	ret = 0;

out:
	freeattr(a);
	keyclose(k);
	return ret;
}

Re: [9fans] how to avoid a memset() optimization

[David Presotto]

There's always waserror/nexterror

Re: [9fans] how to avoid a memset() optimization

[Sam]

ooh, a challenge.  Nothing pleases me more than proving myself
wrong.  be back in a few. ;)

btw - i don't fear the goto.  I use it a lot in small loop
situations to avoid empty for loops and such.  I'll play
with this function and see what I find. ;)

Cheers,

Sam

On Mon, 6
Jan 2003, Russ Cox wrote:

> > you just need a marker to see how far you got. (I'm not even sure
> > what the context of this message is, but every time i see the goto
> > solution to this I have to say something). ;)
>
> the goto seems much clearer to me.
> also easier to edit.  what happens when
> you have to lock A½?  you have to rewrite
> all your case statements!
>
> g% grep goto /sys/games/lib/fortunes |sed 1q
> If you want to go somewhere, goto is the best way to get there. K Thompson
> g%
>
> also you can't always write the code in your if-style,
> but you can always use a goto.  for example, you can't
> rewrite this without goto (or at least, if you do, it ends
> up a lot less clear or a lot more duplicated):
>
> static int
> p9skclient(Conv *c)
> {
> 	char *user;
> 	char cchal[CHALLEN];
> 	uchar secret[8];
> 	char buf[MAXAUTH];
> 	int speakfor, ret;
> 	Attr *a;
> 	Authenticator au;
> 	Key *k;
> 	Ticket t;
> 	Ticketreq tr;
>
> 	ret = -1;
> 	a = nil;
>  	k = nil;
>
> 	/* p9sk1: send client challenge */
> 	if(c->proto == &p9sk1){
> 		c->state = "write challenge";
> 		memrandom(cchal, CHALLEN);
> 		if(convwrite(c, cchal, CHALLEN) < 0)
> 			goto out;
> 	}
>
> 	/* read ticket request */
> 	c->state = "read tickreq";
> 	if(convread(c, buf, TICKREQLEN) < 0)
> 		goto out;
> 	convM2TR(buf, &tr);
>
> 	/* p9sk2: use server challenge as client challenge */
> 	if(c->proto == &p9sk2)
> 		memmove(cchal, tr.chal, CHALLEN);
>
> 	/*
> 	 * find a key.
> 	 *
> 	 * if the user is the factotum owner, any key will do.
> 	 * if not, then if we have a speakfor key,
> 	 * we will only vouch for the user's local identity.
> 	 *
> 	 * this logic is duplicated in p9any.c
> 	 */
> 	user = strfindattr(c->attr, "user");
> 	a = delattr(copyattr(c->attr), "role");
> 	a = addattr(a, "proto=p9sk1");
>
> 	if(strcmp(c->sysuser, owner) == 0){
> 		speakfor = 0;
> 		a = addattr(a, "proto=p9sk1 user? dom=%q", tr.authdom);
> 	}else if(user==nil || strcmp(c->sysuser, user)==0){
> 		speakfor = 1;
> 		a = delattr(a, "user");
> 		a = addattr(a, "proto=p9sk1 user? dom=%q role=speakfor", tr.authdom);
> 	}else{
> 		werrstr("will not authenticate for %q as %q", c->sysuser, user);
> 		goto out;
> 	}
>
> 	for(;;){
> 		c->state = "find key";
> 		k = keyfetch(c, "%A", a);
> 		if(k == nil)
> 			goto out;
>
> 		/* relay ticket request to auth server, get tickets */
> 		strcpy(tr.hostid, strfindattr(k->attr, "user"));
> 		if(speakfor)
> 			strcpy(tr.uid, c->sysuser);
> 		else
> 			strcpy(tr.uid, tr.hostid);
>
> 		c->state = "get tickets";
> 		if(gettickets(&tr, buf, k) < 0)
> 			goto out;
>
> 		convM2T(buf, &t, k->priv);
> 		if(t.num == AuthTc)
> 			break;
>
> 		/* we don't agree with the auth server about the key; try again */
> 		c->state = "replace key";
> 		if((k = keyreplace(c, k, "key mismatch with auth server")) == nil){
> 			werrstr("key mismatch with auth server");
> 			goto out;
> 		}
> 	}
>
> 	/* send second ticket and authenticator to server */
> 	c->state = "write ticket+auth";
> 	memmove(buf, buf+TICKETLEN, TICKETLEN);
> 	au.num = AuthAc;
> 	memmove(au.chal, tr.chal, CHALLEN);
> 	au.id = 0;
> 	convA2M(&au, buf+TICKETLEN, t.key);
> 	if(convwrite(c, buf, TICKETLEN+AUTHENTLEN) < 0)
> 		goto out;
>
> 	/* read authenticator from server */
> 	c->state = "read auth";
> 	if(convread(c, buf, AUTHENTLEN) < 0)
> 		goto out;
> 	convM2A(buf, &au, t.key);
> 	if(au.num != AuthAs || memcmp(au.chal, cchal, CHALLEN) != 0 || au.id != 0){
> 		werrstr("server lies through his teeth");
> 		goto out;
> 	}
>
> 	/* success */
> 	c->attr = addcap(c->attr, c->sysuser, &t);
> 	des56to64((uchar*)t.key, secret);
> 	c->attr = addattr(c->attr, "secret=%.8H", secret);
> 	ret = 0;
>
> out:
> 	freeattr(a);
> 	keyclose(k);
> 	return ret;
> }
>

Re: [9fans] how to avoid a memset() optimization

[Fco.J.Ballesteros]

If you are so in hurry for cycles, you'd better wait for
six months or so and buy the next generation processor.

IMHO, human time is the most expensive resource most of
the times; so, I'd optimize that instead.

Re: [9fans] how to avoid a memset() optimization

[Scott Schwartz]

| If you are so in hurry for cycles, you'd better wait for
| six months or so and buy the next generation processor.

You want me to wait six months because I'm in a hurry?  :-)

| IMHO, human time is the most expensive resource most of
| the times; so, I'd optimize that instead.

Yes, that's the argument for an optimizing compiler: so that the machine
can quicky do the correctness preserving mechanical improvements instead
of me doing them slowly and wrongly.

Re: [9fans] how to avoid a memset() optimization

[Dennis Ritchie]

Minnich remarked

 > I feel the need. Linuxbios no longer works on gcc 3.2 because of some
 > bizarre optimisations that have gone in that break mptable parsing. One
 > person claims that you can no longer count on this:

 > struct x {
 > 	int a;
 > 	int b;
 > };

 > resulting in code in which a and b are laid out in memory in the same
 > order as in the structure (this due to C++). Structure members can get
 > reordered.

 > I can't believe this is true, but someone claims it is and I have not had
 > time to verify it.

Dunno about C++, but C99 still specifies that
structure members are laid out in the order
they are declared.

	Dennis

Re: [9fans] how to avoid a memset() optimization

[Ronald G. Minnich]

On Wed, 13 Nov 2002, Dennis Ritchie wrote:

> Dunno about C++, but C99 still specifies that
> structure members are laid out in the order
> they are declared.

good. I am hoping the person who brought this up was just plain wrong.

thanks

ron

Re: [9fans] how to avoid a memset() optimization

[Rob `Commander' Pike]

> Yep, we need a C that works for its original purpose.

You, sir, are a curmudgeon.

-rob

Re: [9fans] how to avoid a memset() optimization

[C H Forsyth]

	It's not a bug, it's an allowed optimization.

no doubt that's true in this case, but i did think the phrase had
broader application, so i put it in my fortunes file.

Re: [9fans] how to avoid a memset() optimization

[rob pike]

The various new Worlds Of C out there - gcc, C++, C99 - are
all rife with arcana.  C has lost its original vision as a
systems language close to the machine and turned into a vile
tar baby of gotchas and mysteries.

	It's not a bug, it's an allowed optimization.

does in fact sum it up well.  I think there's actually a need
for a C-like language again, since C is pretty much gone from
the earth.  (Even when the compiler resists the needless
complexity, the libraries and ohmygod include files suffice
to bloat and confuse beyond redemption.)  The problem is of
course that only curmudgeons like me feel the need; everyone
else wants the rat's nest, which is why we got it in the first
place.

Sigh.

-rob

Re: [9fans] how to avoid a memset() optimization

[andrey mirtchovski]

How would Limbo do as a systems language?

merely asking, not suggesting: andrey

Re: [9fans] how to avoid a memset() optimization

[Ronald G. Minnich]

On Wed, 13 Nov 2002, rob pike wrote:

> The problem is of course that only curmudgeons like me feel the need;
> everyone else wants the rat's nest, which is why we got it in the first
> place.

I feel the need. Linuxbios no longer works on gcc 3.2 because of some
bizarre optimisations that have gone in that break mptable parsing. One
person claims that you can no longer count on this:

struct x {
	int a;
	int b;
};

resulting in code in which a and b are laid out in memory in the same
order as in the structure (this due to C++). Structure members can get
reordered.

I can't believe this is true, but someone claims it is and I have not had
time to verify it.

People actually ask me, from time to time, why I care that structure
members get laid out as you declare them. !@#$!@#$#@!. Then they say
something like "you should use a systems programing language if you need
that". !@!@#$#!@$#!$#@.

Yep, we need a C that works for its original purpose.

ron

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Ronald G. Minnich wrote:
> struct x {
> 	int a;
> 	int b;
> };
> resulting in code in which a and b are laid out in memory in the same
> order as in the structure (this due to C++). Structure members can get
> reordered.

Not in Standard C.

Re: [9fans] how to avoid a memset() optimization

[Ronald G. Minnich]

On Thu, 14 Nov 2002, Douglas A. Gwyn wrote:

> Ronald G. Minnich wrote:
> > struct x {
> > 	int a;
> > 	int b;
> > };
> > resulting in code in which a and b are laid out in memory in the same
> > order as in the structure (this due to C++). Structure members can get
> > reordered.
>
> Not in Standard C.


hmmm, is gcc 3.2 standard C? :-)

ron

Re: [9fans] how to avoid a memset() optimization

[Clint Olsen]

In article <Pine.LNX.4.44.0211130911030.27108-100000@carotid.ccs.lanl.gov>,
Ronald G. Minnich wrote:
>
> struct x {
> 	int a;
> 	int b;
> };
>
> resulting in code in which a and b are laid out in memory in the same
> order as in the structure (this due to C++). Structure members can get
> reordered.

This would violate the C standard that a pointer to a struct is compatible
with a pointer to its first argument.  That be busted...

-Clint

Re: [9fans] how to avoid a memset() optimization

[William K. Josephson]

On Wed, Nov 13, 2002 at 07:54:48AM -0800, rob pike wrote:
> the earth.  (Even when the compiler resists the needless
> complexity, the libraries and ohmygod include files suffice
> to bloat and confuse beyond redemption.)  The problem is of
> course that only curmudgeons like me feel the need; everyone
> else wants the rat's nest, which is why we got it in the first
> place.

It gets worse: "you must use foo_uint32_t because we don't want
to get burned if the size of the standard types change".  The
amount of time I wasted yesterday all because people insist using
poorly thought out additions to the language, header files, and
libraries is not to be believed.  They mumble about "simplicity"
and then insist on using the worst voodoo in C99 :-)

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

William K. Josephson wrote:
> It gets worse: "you must use foo_uint32_t because we don't want
> to get burned if the size of the standard types change".  The
> amount of time I wasted yesterday all because people insist using
> poorly thought out additions to the language, header files, and
> libraries is not to be believed.  They mumble about "simplicity"
> and then insist on using the worst voodoo in C99 :-)

There is no foo_* in Standard C.
If you're referring to the <stdint.h> typedefs, those are
available so that programmers who *need* to be somewhat
precise about their integer widths have a standard way
to do so.  How much code have you had to port where it
was assumed that long was 32 bits, or 4 chars, or that
short was aliasable to an array of 2 chars?  We had the
dickens of a time getting such code fixed to work on
64-bit platforms.
If you're referring to such things as off_t needing to
be enlarged as a system evolves, applications don't need
to use anything other than off_t at the source level.
The only problem occurs at the binary interface level.
Vendors such as Sun who have had to deal with binary
compatibility in the face of types widening can explain
the problems and some of the (kludge) methods of
addressing them; there really isn't anything that could
have been done at the source level to prevent that problem.

Re: [9fans] how to avoid a memset() optimization

[William Josephson]

On Thu, Nov 14, 2002 at 10:21:56AM +0000, Douglas A. Gwyn wrote:
> William K. Josephson wrote:
> >It gets worse: "you must use foo_uint32_t because we don't want
> >to get burned if the size of the standard types change".  The
> >amount of time I wasted yesterday all because people insist using
> >poorly thought out additions to the language, header files, and
> >libraries is not to be believed.  They mumble about "simplicity"
> >and then insist on using the worst voodoo in C99 :-)
>
> There is no foo_* in Standard C.

You are completely missing the point.

> to do so.  How much code have you had to port where it
> was assumed that long was 32 bits, or 4 chars, or that
> short was aliasable to an array of 2 chars?  We had the
> dickens of a time getting such code fixed to work on
> 64-bit platforms.

Plenty; that's not the point.

> If you're referring to such things as off_t needing to
> be enlarged as a system evolves, applications don't need
> to use anything other than off_t at the source level.

No again.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

rob pike wrote:
>     It's not a bug, it's an allowed optimization.
> does in fact sum it up well.  I think there's actually a need
> for a C-like language again, since C is pretty much gone from
> the earth.  (Even when the compiler resists the needless
> complexity, the libraries and ohmygod include files suffice
> to bloat and confuse beyond redemption.)  The problem is of
> course that only curmudgeons like me feel the need; everyone
> else wants the rat's nest, which is why we got it in the first
> place.

Actually I argued in committee for requiring all indicated
accesses without requiring introduction of volatile
qualification, but essentially all the compiler vendors
said that their customers typically preferred faster
(optimized) code and would reject any C standard (thus
standard-conforming compilers) that disallowed whatever
optimizations could be thought up that didn't change the
outcome of the computation.  At least with the introduction
of volatile qualification there is a standard way for the
programmer to obtain the specific behavior desired in this
case.

C as you imagine it was never really in existence; even on
the PDP-11, certain access patterns (such as dereference
of constants in the range 0160000-017777t) were specially
(kludge) excluded from optimization, so that device drivers
wouldn't be broken by non-1-to-1 translations from source
code to accesses.  "volatile" addresses such issues as
well as we could with a simple portable mechanism.

The main reason for the growth of library functios and
include files is that it is relatively easy to add new
features, but once enough people have developed applications
that depend on them, it is horribly painful and expensive
to remove the features.

In the case of duplicate sets of character/string functions,
I largely lay the blame for that on Dennis's public insistence
that it was fundamental to C that sizeof(char)==1.  That in
effect guaranteed that the same size object could not be used
both for basic storage unit and for character code, so the
traditional str* functions immediately became inadequate.
It would have been nice to junk them, but obviously they
were too widely used by then and had to be standardized.

We do the best we can in the face of conflicting requirements.

Re: [9fans] how to avoid a memset() optimization

[Dan Cross]

> Actually I argued in committee for requiring all indicated
> accesses without requiring introduction of volatile
> qualification, but essentially all the compiler vendors
> said that their customers typically preferred faster
> (optimized) code and would reject any C standard (thus
> standard-conforming compilers) that disallowed whatever
> optimizations could be thought up that didn't change the
> outcome of the computation.  At least with the introduction
> of volatile qualification there is a standard way for the
> programmer to obtain the specific behavior desired in this
> case.

I don't have my copy of C99 handy, but can you put volatile
in a function's prototype as a hint to the compiler that calls
to that function should *not* be optimized away?  It seems
like that, combined with, e.g., smemset() would solve the
problem.  smemset(), from a design perspective, really is the
right approach, and expecting inexperienced programmers to
understand all the subtleties of volatile aren't.  Example:
``what's this volatile thing do?  Wow; I deleted it and the
code still compiles.  And it's much nicer looking without it.
(hum hum hum)''

> In the case of duplicate sets of character/string functions,
> I largely lay the blame for that on Dennis's public insistence
> that it was fundamental to C that sizeof(char)==1.  That in
> effect guaranteed that the same size object could not be used
> both for basic storage unit and for character code, so the
> traditional str* functions immediately became inadequate.
> It would have been nice to junk them, but obviously they
> were too widely used by then and had to be standardized.

I personally think that C should have a byte type, and not
char.  Char is really about bytes masquerading as characters,
not character data.  Dennis is right however, that you need
a type with sizeof(that_type) == 1.

> We do the best we can in the face of conflicting requirements.

Sometimes the best you can do creates more conflict, though.  Not
a dig, but a truism.

	- Dan C.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Dan Cross wrote:
> I don't have my copy of C99 handy, but can you put volatile
> in a function's prototype as a hint to the compiler that calls
> to that function should *not* be optimized away?

Yes, if declared properly, the accesses through the pointer to
volatile objects would be required to occur, although other
optimizations (such as in-lining) could still be performed.

One could in principle cast non-volatile object references to
volatile pointers when invoking a function, but if the prototype
didn't include the volatile qualifer it is currently a constraint
violation.

> smemset(), from a design perspective, really is the right approach

memset((volatile void*)p,n,0) would be similar if it were allowed.

> I personally think that C should have a byte type, and not
> char.  Char is really about bytes masquerading as characters,
> not character data.  Dennis is right however, that you need
> a type with sizeof(that_type) == 1.

Oh, I agree.  In fact my 1986 proposal chose "short char" as the
name for the byte type in order not to introduce a new keyword.
The only change to applications would be for malloc invocations
for char arrays to be modified to match the usage style for other
arrays: p = malloc(n*sizeof(char)); in fact at around that time I
adopted that style for my own programming and it wasn't a lot of
trouble.

One advantage to separating the storage unit size from particular
data types is that it supports bit-addressable architectures,
where short char might occupy 1 bit.  I've often needed bit arrays
and it has been a pity that the language wouldn't support an
efficient allocation of them.  Indeed there has been a chicken-and-
egg cycle here; one computer architect contacted me hoping that
the forthcoming C standard would offer such support, since he was
agitating for bit addressability in a new architecture in the works.
Unfortunately, he was met by the objection "If we had that feature
it would not be available to the HLL programmer anyway."

My general feeling is that it is way too late to "fix" C, but a
C-like language could be designed to take into account what we
should have learned from actual C experience.  However, I don't
know who I would trust to get it right.

Re: [9fans] how to avoid a memset() optimization

[Tad Hunt]

someone mentioned something along the lines of

void*
secmemset(void *buf, int v, int len)
{
	return memset((volatile void*)buf, v, len);
}

However, isn't GCC going to complain that you're passing
a volatile to a function that isn't expecting a volatile?

-Tad

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Tad Hunt wrote:
> void*
> secmemset(void *buf, int v, int len)
> {
>         return memset((volatile void*)buf, v, len);
> }
> However, isn't GCC going to complain that you're passing
> a volatile to a function that isn't expecting a volatile?

In standard-conforming mode it is *obliged* to issue a
diagnostic.  However, it can (and probably does) still
generate code.  Since this diagnostic would occur only
once (when putting secmemset into some library) it
wouldn't be as big a deal as if it occurred for every
program that used secmemset.

Re: [9fans] how to avoid a memset() optimization

[Aharon Robbins]

In article <3DD3D463.BE431AC7@arl.army.mil> you write:
>My general feeling is that it is way too late to "fix" C, but a
>C-like language could be designed to take into account what we
>should have learned from actual C experience.  However, I don't
>know who I would trust to get it right.

Why, Dennis Ritchie, of course.

What would be really cool would be to do it as front end to GCC.
Then we'd get portability to all the architectures it supports in
one short.

AND we'd get all the nifty optimisations that GCC does, for free!

Oh! ... Wait a minute....  Never mind.

:-)

Arnold

Re: [9fans] how to avoid a memset() optimization

[Skip Tavakkolian]

> How would Limbo do as a systems language?

Limbo does things for you that one wouldn't want in a systems
language, and neither is Alef, although the name would fit.

Nowadays do we need anything more than a Dis translator?  Everyone
could learn Dis or MMIX (Knuth's) and be done with it.  After all,
portability was the main goal.

Maybe Forth is the answer ☺

Re: [9fans] how to avoid a memset() optimization

[rog]

> For one thing, it doesn't work (unless you force
> volatile qualification within secmemset)

which seems reasonable to me.

and for barely more runtime cost, one could define:

void*
secmemset(void *ap, int c, ulong n)
{
	memset((volatile void *)ap, c, n);
}

(or is that "void * volatile"?  i never sussed this C type qualifier
stuff).

is that not sufficient to stymie any legitimate optimisations that GCC
might try to do, while maintaining comparable performance?

(not to mention making sure that you don't kill the performance of any
crypto tight loops that might be running on the original buffer by
declaring it volatile).

  cheers,
    rog.

Re: [9fans] how to avoid a memset() optimization

[Skip Tavakkolian]

> gcc(1) is a treasure hunt; there's all sorts of stuff buried in there.

There should be some sort of certification one could get for
knowing all gcc options.

On the issue of optimization, I think if the programmer has to figure
out what the compiler has done through exegesis or inference, then
there is something wrong with the compiler or the specification;

I feel I should only be required to know a compiler the way I know my
car.  I expect it to behave according to the manual.  I shouldn't be
required to know a compiler the way I know a close family member.

Re: [9fans] how to avoid a memset() optimization

[Boyd Roberts]

Skip Tavakkolian wrote:

>There should be some sort of certification one could get for
>knowing all gcc options.
>
>
If that were possible, you would be certifiable.

Re: [9fans] how to avoid a memset() optimization

[C H Forsyth]

>>No, it mens "compiler, the access must really be
>>performed".  It is the programmer who decides whether
>>that is a requirement, and it could be for numerous
>>reasons, such as to assist in debugging, because the
"location" is a portal to an external device register,
>>because the object is shared among threads, etc.  In
>>this particular case it would be because the programmer
>>really wants the memory overwrite to occur regardless
>>of whether it is used any further.

so, in the case of memset applied to such a thing,
are those accesses that `must really be performed'
byte accesses, word accesses, quadword?

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

C H Forsyth wrote:
> so, in the case of memset applied to such a thing,
> are those accesses that `must really be performed'
> byte accesses, word accesses, quadword?

The indicated bytes must all end up with 0 values,
which was the important thing anyway for this application.
The width of actual access depends on the implementation.

Re: [9fans] how to avoid a memset() optimization

[Geoff Collyer]

When I'm stuck with using gcc, I invoke it with this script:

#!/bin/sh
# agcc - invoke gcc in ansi mode
# the --writable-strings is for pre-ansi programs like some versions of mk
exec gcc -ansi -pipe -fvolatile --writable-strings "$@"

-fvolatile makes gcc treat all data as if declared volatile.  I
sometimes add (usually to the ?akefile) `-Dconst=' to deal with const
poisoning.

gcc(1) is a treasure hunt; there's all sorts of stuff buried in there.

Re: [9fans] how to avoid a memset() optimization

[Boyd Roberts]

Geoff Collyer wrote:

>gcc(1) is a treasure hunt; there's all sorts of stuff buried in there.
>
s/treasure hunt/minefield/

[9fans] how to avoid a memset() optimization

[Andrew Simmons]

>We don't optimize out subroutine calls that are in live
>codd, mostly because they might have side effects.

A wise policy. I tend to avoid all subroutine calls involving live fish,
especially the tench, roach, bream, and, most dangerous of all, the gudgeon.

Re: [9fans] how to avoid a memset() optimization

[Doc Shipley]

On Wed, 13 Nov 2002, Andrew Simmons wrote:

> >We don't optimize out subroutine calls that are in live
> >codd, mostly because they might have side effects.
>
> A wise policy. I tend to avoid all subroutine calls involving live fish,
> especially the tench, roach, bream, and, most dangerous of all, the gudgeon.

  I can see that you were in high gudgeon when you read that post....

	Doc

Re: [9fans] how to avoid a memset() optimization

[Russ Cox]

>   Putting aside issues with optimizing compilers, I don't see a "volatile"
>   qualifier as something that "hurts readability and clarity", I even
>   would say, that it is exactly the case where its use is justified.
>   Wouldn't you agree ?

My problem with volatile is that it means ``something
weird is going on here'' but doesn't tell you what.
It's also not near the weirdness.  Using smemset says
``this is a security-critical memset; it cannot get
optimized away.''  Otherwise I just see a memset
of a buffer that happens to be volatile.  What does
that tell me?  Very little.  And the volatile tag might
be miles away from the memset call, making it even
easier to forget that they're related.

>   You're right here, but for me defining secmemset is much bigger evil
>   than just adding volatile to the buffer.

Why?

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Russ Cox wrote:
> My problem with volatile is that it means ``something
> weird is going on here'' but doesn't tell you what.

No, it mens "compiler, the access must really be
performed".  It is the programmer who decides whether
that is a requirement, and it could be for numerous
reasons, such as to assist in debugging, because the
"location" is a portal to an external device register,
because the object is shared among threads, etc.  In
this particular case it would be because the programmer
really wants the memory overwrite to occur regardless
of whether it is used any further.

>> You're right here, but for me defining secmemset
 >> is much bigger evil than just adding volatile to
 >> the buffer.
> Why?

For one thing, it doesn't work (unless you force
volatile qualification within secmemset), because
the invocation of secmemset can be optimized away
also.  (It is less likely than for memset, but it
is a possibility when the compiler can see the
caller and callee.)  Also you're in effect just
replacing the standard library function with your
own implementation, which will in general be slower
(a trade-off you might be willing to make) and one
more interface to specify (assuming a professional
software environment as opposed to just hacking).
In general one should have better things to do than
duplicating standard library functions.

Re: [9fans] how to avoid a memset() optimization

[Roman V. Shaposhnick]

On Tue, Nov 12, 2002 at 08:47:41PM -0500, Russ Cox wrote:
> >   You're right here, but for me defining secmemset is much bigger evil
> >   than just adding volatile to the buffer.
>
> Why?

  Because dropping volatile creates more problems in subsequent code.
  Inexperienced maintainer might add some new code and end it with
  a call to memset() [ he is inexperienced and doesn't know about
  secmemset yet ] and he will trip over without buffer being volatile.

  For me this is worse.

Thanks,
Roman.

Re: [9fans] how to avoid a memset() optimization

[William Josephson]

On Thu, Nov 14, 2002 at 04:46:45AM +0300, Roman V. Shaposhnick wrote:
>   Because dropping volatile creates more problems in subsequent code.
>   Inexperienced maintainer might add some new code and end it with
>   a call to memset() [ he is inexperienced and doesn't know about
>   secmemset yet ] and he will trip over without buffer being volatile.
>
>   For me this is worse.

If it is old code, then he'll see the secmemset call and
go read up on it.  If it is new code and doesn't know about
secmemset, what is the chance he will know to use volatile?

Re: [9fans] how to avoid a memset() optimization

[Roman V. Shaposhnick]

On Wed, Nov 13, 2002 at 08:52:57PM -0500, William Josephson wrote:
> On Thu, Nov 14, 2002 at 04:46:45AM +0300, Roman V. Shaposhnick wrote:
> >   Because dropping volatile creates more problems in subsequent code.
> >   Inexperienced maintainer might add some new code and end it with
> >   a call to memset() [ he is inexperienced and doesn't know about
> >   secmemset yet ] and he will trip over without buffer being volatile.
> >
> >   For me this is worse.
>
> If it is old code, then he'll see the secmemset call and
> go read up on it.  If it is new code and doesn't know about
> secmemset, what is the chance he will know to use volatile?

  He doesn't need to know about volatile -- that's the whole point.
  Once "experienced" programmer has set the buffer that way, the
  less experienced just gets less chance to screw himself up
  ( although, the right to screw yourself is probably one of the
  basic human rights, so it's not eliminated here either ).

  With the secmemset approach the chance is greater.

Thanks,
Roman.

Re: [9fans] how to avoid a memset() optimization

[Russ Cox]

> Plan9 has its own C compiler ) but seeing how gcc is making its
> way, it seems that declaring every buffer volatile ( as John the
> moderator has suggested ) wouldn't hurt either.

I disagree.  It hurts readability and clarity, all
to avoid a bug in gcc.  If I asked it to zero the
memory, then, damn it!, zero the memory.

The right fix is to avoid gcc.

After accepting the fact that gcc cannot be avoided,
cannot be fixed everywhere at once, and might not
ever be fixed anywhere at all, the right fix is to declare
a ``secure memset'' function:

	void*
	smemset(void *ap, int c, ulong n)
	{
		char *p;

		p = ap;
		while(n > 0) {
			*p++ = c;
			n--;
		}
		return ap;
	}

The only difference between this memset and the
standard portable libc memset is the name.
The point is that this only happens because gcc
is inlining the memset and then treating it as a
sequence of unused assignments.  Gcc ``knows'' too
much about memset.  If you call it something else,
that special knowledge goes away, as does the bug,
especially if smemset is in its own source file.

Russ

Re: [9fans] how to avoid a memset() optimization

[Roman V. Shaposhnick]

On Tue, Nov 12, 2002 at 07:31:13PM -0500, Russ Cox wrote:
> > Plan9 has its own C compiler ) but seeing how gcc is making its
> > way, it seems that declaring every buffer volatile ( as John the
> > moderator has suggested ) wouldn't hurt either.
>
> I disagree.  It hurts readability and clarity, all
> to avoid a bug in gcc.

  Putting aside issues with optimizing compilers, I don't see a "volatile"
  qualifier as something that "hurts readability and clarity", I even
  would say, that it is exactly the case where its use is justified.
  Wouldn't you agree ?

> If I asked it to zero the memory, then, damn it!, zero the memory.
> The right fix is to avoid gcc.

  gcc is not the only one to blame, most of the compilers which pride
  themselves on doing witty optimizations are guilty as well. The
  point here is that "volatile" seems to be the only portable
  way of telling optimizer: "get away from this data".

> The point is that this only happens because gcc
> is inlining the memset

  You're right here, but for me defining secmemset is much bigger evil
  than just adding volatile to the buffer. The problem with volatile, of
  course, is that it is just a hint not a directive ( well, at least
  that's how I interpret C99 Standard ).

Thanks,
Roman.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Roman V. Shaposhnick wrote:
> The problem with volatile, of course, is that it is just a
 > hint not a directive ( well, at least  that's how I interpret
 > C99 Standard ).

I don't see how you can interpret the C standard that way.
Volatile qualification imposes a *requirement* that the
accesses of the abstract C machine model actually be
performed in the generated code.  Details of the access,
such as width, read-modify-write vs. write, etc. must be
documented by the C implementation.

Re: [9fans] how to avoid a memset() optimization

[Roman V. Shaposhnick]

On Wed, Nov 13, 2002 at 10:15:56AM +0000, Douglas A. Gwyn wrote:
> Roman V. Shaposhnick wrote:
> > The problem with volatile, of course, is that it is just a
>  > hint not a directive ( well, at least  that's how I interpret
>  > C99 Standard ).
>
> I don't see how you can interpret the C standard that way.

I don't have C99 standard handy, but here's what C++ ISO Standard
says about volatile ( para 7.1.5.1 section 8 ):

"Note: volatile is a hint to the implementation to avoid aggressive
optimization involving the object because the value of the object
might be changed by means undetectable by an implementation. "

and since later on in the same section we read:

"In general, the semantics of volatile are intended to be the same in
C++ as they are in C."

I assumed that for C99 it is also merely a hint. I'd be glad to
be contradicted with factual information.

Thanks,
Roman.

Re: [9fans] how to avoid a memset() optimization

[Douglas A. Gwyn]

Russ Cox wrote:
> I disagree.  It hurts readability and clarity, all
> to avoid a bug in gcc.  If I asked it to zero the
> memory, then, damn it!, zero the memory.

It's not a bug, it's an allowed optimization (in the
absence of volatile qualification).  The question is
where to draw the line for degree of optimization,
and the volatile qualifier was invented precisely to
draw the line at actually performing the coded
accesses to the variable whether or not the result
is further used in the program.

Re: [9fans] how to avoid a memset() optimization

[presotto]

[-- Attachment #1: Type: text/plain, Size: 163 bytes --]

We don't optimize out subroutine calls that are in live
codd, mostly because they might have side effects.  We
do optimize out expressions that don't get called.

[-- Attachment #2: Type: message/rfc822, Size: 3132 bytes --]

From: "Roman V. Shaposhnick" <vugluskr@unicorn.math.spbu.ru>
To: 9fans@cse.psu.edu
Subject: [9fans] how to avoid a memset() optimization
Date: Wed, 13 Nov 2002 01:42:17 +0300
Message-ID: <20021113014217.A5718@unicorn.math.spbu.ru>

There's quite a lively discussion on comp.compilers about memset()
use in security related applications and compiler optimization.
This idiom of memset'ing secure stuff with 0 is not new, and I've
spotted it in Plan9 sources several times -- don't know whether
it should be a concern for Plan9 users and developers ( after all,
Plan9 has its own C compiler ) but seeing how gcc is making its
way, it seems that declaring every buffer volatile ( as John the
moderator has suggested ) wouldn't hurt either.

Thanks,
Roman.

From: "Francis Wai" <fwai@rsasecurity.com>
Newsgroups: comp.compilers
Subject: how to avoid a memset() optimization
Date: 7 Nov 2002 00:51:51 -0500

In a recent article (http://online.securityfocus.com/archive/82/297827),
Peter Gutmann raised a concern which has serious implications in
secure programming. His example, along the lines of,

int main()
{
    char key[16];
    strcpy(key, "whatever");
    encrpts(key);
    memset(key, 0, 16);
}

where memset() was optimized away because memset() is the last
expression before the next sequence point and that its side-effect is
not needed and that the subject of memset() is an auto variable. The
compiler sees that it is legitimate to optimize it away. This is _bad_
news for anyone concerns with sensitive data being left lying around
in memory.

Various suggestions have been made, such as declaring the variable
volatile and having a scrub memory function in a file of its own. I'm
wondering if there are better ways such as telling the compiler not to
optimize away a function call.
[Declaring the array volatile is the right way to do it.  The reason
volatile exists is to tell the compiler not to do otherwise valid
optimizations. -John]

[9fans] how to avoid a memset() optimization

[Roman V. Shaposhnick]

There's quite a lively discussion on comp.compilers about memset()
use in security related applications and compiler optimization.
This idiom of memset'ing secure stuff with 0 is not new, and I've
spotted it in Plan9 sources several times -- don't know whether
it should be a concern for Plan9 users and developers ( after all,
Plan9 has its own C compiler ) but seeing how gcc is making its
way, it seems that declaring every buffer volatile ( as John the
moderator has suggested ) wouldn't hurt either.

Thanks,
Roman.

From: "Francis Wai" <fwai@rsasecurity.com>
Newsgroups: comp.compilers
Subject: how to avoid a memset() optimization
Date: 7 Nov 2002 00:51:51 -0500

In a recent article (http://online.securityfocus.com/archive/82/297827),
Peter Gutmann raised a concern which has serious implications in
secure programming. His example, along the lines of,

int main()
{
    char key[16];
    strcpy(key, "whatever");
    encrpts(key);
    memset(key, 0, 16);
}

where memset() was optimized away because memset() is the last
expression before the next sequence point and that its side-effect is
not needed and that the subject of memset() is an auto variable. The
compiler sees that it is legitimate to optimize it away. This is _bad_
news for anyone concerns with sensitive data being left lying around
in memory.

Various suggestions have been made, such as declaring the variable
volatile and having a scrub memory function in a file of its own. I'm
wondering if there are better ways such as telling the compiler not to
optimize away a function call.
[Declaring the array volatile is the right way to do it.  The reason
volatile exists is to tell the compiler not to do otherwise valid
optimizations. -John]

Re: [9fans] useful language extension, or no?

[rob pike, esq.]

Saves you, what, 5 keystrokes?  Not too convincing.

-rob

Re: [9fans] useful language extension, or no?

[Sam]

You're right, and the capability to do:

struct type {
	int a;
	int b;
};

struct type2 {
	type;
	int c;
	int d;
};

saves so many more strokes in itself *and must* be more efficient.

Suppose I'm not saying "why," but "why not."  IMO it's cleaner and
quite possibly more efficient (without getting into a usec argument,
please).  Do you disagree?

Sam

On Tue, 16 Jul 2002, rob pike, esq. wrote:

> Saves you, what, 5 keystrokes?  Not too convincing.
>
> -rob
>

Re: [9fans] useful language extension, or no?

[Steve Kilbane]

> Suppose I'm not saying "why," but "why not."  IMO it's cleaner

I disagree. It's considerably more ambiguous. Your loop continuation
condition (i<NELS) may be failed by considerably more cases
than your trailing condition (i==NELS). What happens if your loop
body does "i=NELS+1"? What if your "i++" was actually something
like "i=arr[i]"? What if your continuation condition was
"i<NELS && j<NELS2"?

C's "for" is not like Pascal's FOR.

> and
> quite possibly more efficient

The pros and cons have already been discussed. I'd just like to
note that there's no particularly compelling reason why it would
generate different code from that which used the idiom. A good
optimiser would render them into the same internal form anyway.

"A good optimiser" is a subjective term, and Plan 9's C optimising
has already been discussed recent (Summary: it's good enough that
there are more important/interesting/useful things to work on),
but the point stands.

steve