x509.c for ape in libsec.a

x509.c for ape in libsec.a

[kokamoto]

I'm playing with python2.7.6 on 9front.
The ape/libsec.a file for this port makes x509-ape.c instead of x509.c on the fly by the line:

x509-ape.c:	../../../../libsec/port/x509.c
	cat ../../../../libsec/port/x509.c | \
	sed 's/Tm \*tm = gmtime\(t\)/struct tm *tm0 = gmtime\(\&t\)/;s/tm->/tm0->tm_/g' | \
	sed 's/static //g' > x509-ape.c

Essential part is 's/static //g' to use is_seq etc ib cpython/Plan9/_ssl9.c.
Is this OK? for 9front system?

Kenji

PS: I hve permission to port python2.7.6 to 9front from jas.
Porting 2.7.6 is not so bright because it has no implementation of _ctypes() module.
So, this just a play...

Re: [9front] x509.c for ape in libsec.a

[kokamoto]

Thank you very much for your kind silence.☺

I solved it locally by making suitable x509-ape.c, and
compiled it Ok.

Now it reports:

term% PYTHONHOME=/sys/lib/python2.7
term% python2.7
Python 2.7.9 (v2.7.8: unknown, Jul 2 2020 12:38:14)[C] on plan9
Type "help", "copyright", "credits" or "license" for more information

I still have complie Objects/unicodeobject.c, stringobject.c, then I
copied thos .8 files from 9legacy machine.

Kenji

Re: [9front] x509.c for ape in libsec.a

[Ori Bernstein]

On Thu, 2 Jul 2020 13:54:11 +0900, kokamoto@hera.eonet.ne.jp wrote:

> 
> I still have complie Objects/unicodeobject.c, stringobject.c, then I
> copied thos .8 files from 9legacy machine.

What were the problems you ran into compiling them?

-- 
    Ori Bernstein

Re: [9front] x509.c for ape in libsec.a

[kokamoto]

> What were the problems you ran into compiling them?

Sorry, I copied wrong cpython source tree from that I compiled
on 9legacy, which is python2.7.6 not 2.7.9.

Kenji

Re: [9front] x509.c for ape in libsec.a

[kokamoto]

I recopied wright python2.7.6 source tree, and have the same
problem so on.

stringobject.c and unicodeobject.c

The errors may be related to cpp, such like:

substargs()->maketokenrow()->domalloc()->malloc()->poolalloc()->p->lock(p);

acid of cpp for those files indivcate different things:

for stringobject.c:
	 static PyObject *
	SubString_new_object_or_empty(SubString *str)
	{
	    if (str->ptr ==((void *)0)) {
	        return PyString_FromStringAndSize(((void *)0), 0);
	    }
	    return PyString_FromStringAndSize(str->ptr, str->end - str->cpp 612: suicide: sys: trap: fault write addr=0xdeffeff4 pc=0x00008705

for unicodeobject.c:
	static PyObject *
	getitem_sequence(PyObject *obj, Py_ssize_t idx)
	{
	    return PySequence_GetItem(obj, idx);
	}


Kenji

static PyObjecpp 766: suicide: sys: trap: fault write addr=0xdeffeff4 pc=0x00008705

>> What were the problems you ran into compiling them?
> 
> Sorry, I copied wrong cpython source tree from that I compiled
> on 9legacy, which is python2.7.6 not 2.7.9.

Re: [9front] x509.c for ape in libsec.a

[kokamoto]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

More precisely, 
for stringobject.c:
	 static PyObject *
	SubString_new_object_or_empty(SubString *str)
	{
	    if (str->ptr ==((void *)0)) {
	        return PyString_FromStringAndSize(((void *)0), 0);
	    }
	    return PyString_FromStringAndSize(str->ptr, str->end - str->cpp 612: suicide: sys: trap: fault write addr=0xdeffeff4 pc=0x00008705

The last line shows PyString_FromStringAndSize() function in cpython/Objects/stringobject.c,
and to do return (PyObject *) op;
Probably, this 'op' is not on the memory...

I replaced /386/bin/cpp by that from 9legacy'e one, and recompiled.
Yes, it's Ok with no problem.

Therefore 9front's /386/bin/cpp has some problem, I think.

Kenji

[-- Attachment #2: Type: message/rfc822, Size: 3158 bytes --]

From: kokamoto@hera.eonet.ne.jp
To: 9front@9front.org
Subject: Re: [9front] x509.c for ape in libsec.a
Date: Wed, 8 Jul 2020 14:19:17 +0900
Message-ID: <702D55686C960FDA9FFE71F48D57ADAA@hera.eonet.ne.jp>

I recopied wright python2.7.6 source tree, and have the same
problem so on.

stringobject.c and unicodeobject.c

The errors may be related to cpp, such like:

substargs()->maketokenrow()->domalloc()->malloc()->poolalloc()->p->lock(p);

acid of cpp for those files indivcate different things:

for stringobject.c:
	 static PyObject *
	SubString_new_object_or_empty(SubString *str)
	{
	    if (str->ptr ==((void *)0)) {
	        return PyString_FromStringAndSize(((void *)0), 0);
	    }
	    return PyString_FromStringAndSize(str->ptr, str->end - str->cpp 612: suicide: sys: trap: fault write addr=0xdeffeff4 pc=0x00008705

for unicodeobject.c:
	static PyObject *
	getitem_sequence(PyObject *obj, Py_ssize_t idx)
	{
	    return PySequence_GetItem(obj, idx);
	}


Kenji

static PyObjecpp 766: suicide: sys: trap: fault write addr=0xdeffeff4 pc=0x00008705

>> What were the problems you ran into compiling them?
> 
> Sorry, I copied wrong cpython source tree from that I compiled
> on 9legacy, which is python2.7.6 not 2.7.9.

Re: [9front] x509.c for ape in libsec.a

[ori]

> More precisely, 
> for stringobject.c:
> 	 static PyObject *
> 	SubString_new_object_or_empty(SubString *str)
> 	{
> 	    if (str->ptr ==((void *)0)) {
> 	        return PyString_FromStringAndSize(((void *)0), 0);
> 	    }
> 	    return PyString_FromStringAndSize(str->ptr, str->end - str->cpp 612: suicide: sys: trap: fault write addr=0xdeffeff4 pc=0x00008705
> 
> The last line shows PyString_FromStringAndSize() function in cpython/Objects/stringobject.c,
> and to do return (PyObject *) op;
> Probably, this 'op' is not on the memory...
> 
> I replaced /386/bin/cpp by that from 9legacy'e one, and recompiled.
> Yes, it's Ok with no problem.
> 
> Therefore 9front's /386/bin/cpp has some problem, I think.
> 
> Kenji

Yes. I'll look into it.

Re: [9front] x509.c for ape in libsec.a

[kokamoto]

This seems to be so from very old, the timestamp of Feb 1, 2018
does fail, too.

Kenji

>> More precisely, 
>> for stringobject.c:
>> 	 static PyObject *
>> 	SubString_new_object_or_empty(SubString *str)
>> 	{
>> 	    if (str->ptr ==((void *)0)) {
>> 	        return PyString_FromStringAndSize(((void *)0), 0);
>> 	    }
>> 	    return PyString_FromStringAndSize(str->ptr, str->end - str->cpp 612: suicide: sys: trap: fault write addr=0xdeffeff4 pc=0x00008705
>> 
>> The last line shows PyString_FromStringAndSize() function in cpython/Objects/stringobject.c,
>> and to do return (PyObject *) op;
>> Probably, this 'op' is not on the memory...
>> 
>> I replaced /386/bin/cpp by that from 9legacy'e one, and recompiled.
>> Yes, it's Ok with no problem.
>> 
>> Therefore 9front's /386/bin/cpp has some problem, I think.
>> 
>> Kenji
> 
> Yes. I'll look into it.

Re: [9front] x509.c for ape in libsec.a

[kokamoto]

Correction!

> This seems to be so from very old, the timestamp of Feb 1, 2018
> does fail, too.

Actually, this fails only if I use cpp by object=386 from cpu (amd64) machine.

If I login on a real 386 terminal, it does success.

In this case, /386/cpp with Oct 5, 2019 time stamp does right
thing, but that with Jun 17, 2020 does fail.

Kenji

PS: sorry confusion☺

Re: [9front] x509.c for ape in libsec.a

[ori]

> This seems to be so from very old, the timestamp of Feb 1, 2018
> does fail, too.
> 
> Kenji
> 
>>> More precisely, 
>>> for stringobject.c:
>>> 	 static PyObject *
>>> 	SubString_new_object_or_empty(SubString *str)
>>> 	{
>>> 	    if (str->ptr ==((void *)0)) {
>>> 	        return PyString_FromStringAndSize(((void *)0), 0);
>>> 	    }
>>> 	    return PyString_FromStringAndSize(str->ptr, str->end - str->cpp 612: suicide: sys: trap: fault write addr=0xdeffeff4 pc=0x00008705
>>> 
>>> The last line shows PyString_FromStringAndSize() function in cpython/Objects/stringobject.c,
>>> and to do return (PyObject *) op;
>>> Probably, this 'op' is not on the memory...
>>> 
>>> I replaced /386/bin/cpp by that from 9legacy'e one, and recompiled.
>>> Yes, it's Ok with no problem.
>>> 
>>> Therefore 9front's /386/bin/cpp has some problem, I think.
>>> 
>>> Kenji
>> 
>> Yes. I'll look into it.

That's surprising, honestly -- I did a lot of surgery on cpp recently
to make it much more standard compliant, and I wouldn't be surprised if
I missed a subtle case.

What's happening right now is that we're recursing infinitely when we
expand *something*.

I'm not sure what yet, but I can reproduce it, so I should be able to
get a fix from the information I have.

I'm guessing that the fix will be shuffling around the way that we
build up our hide-sets.

Re: [9front] x509.c for ape in libsec.a

[Dr. Kenji Okamoto]

 >This seems to be so from very old, the timestamp of Feb 1, 2018
 >does fail, too.

This is not true, it must be said if I do objtype=386 mk from amd64 machine.

If I login to a real 386 terminal, it does success.
The cpp time stamped Oct 7, 2019 does right thing, but that of Jul 17, 
2020 fails.

Now, my file/cpu sever went into partially fail, I'm writing this mail 
from Ubuntu.

Kenji

PS: Ori please don't send me two same mail, when I write reply mail from 
acme,
I cannot distinguish the two. I replied to you directly, which was not 
my intention.

Re: [9front] x509.c for ape in libsec.a

[kokamoto]

> The cpp time stamped Oct 7, 2019 does right thing, but that of Jul 17, 
> 2020 fails.
s/Jul 17 2020/Jun 17 2020/

sorry

Kenji

Re: [9front] x509.c for ape in libsec.a

[ori]

>> The cpp time stamped Oct 7, 2019 does right thing, but that of Jul 17, 
>> 2020 fails.
> s/Jul 17 2020/Jun 17 2020/
> 
> sorry
> 
> Kenji

Alright, I think I have a fix for this issue. We need to put the
macro into the hideset before doing the final pass, or there's
potential for macro expansion to cycle in some cases.

	expand(A) => B
	expand(B) => A
	expand(A) => B
	...

This diff seems to work, but it's late and it needs more testing.
I've run it against our base system, netsurf, and perl.

diff -r 7bd3f3b9dc76 sys/src/cmd/cpp/macro.c
--- a/sys/src/cmd/cpp/macro.c	Sun Jul 05 22:15:02 2020 +0200
+++ b/sys/src/cmd/cpp/macro.c	Thu Jul 09 21:10:36 2020 -0700
@@ -169,15 +169,32 @@
 		}
 		if (np->flag&ISMAC)
 			builtin(trp, np->val);
-		else {
+		else
 			expand(trp, np);
-		}
 		tp = trp->tp;
 	}
 	if (flag)
 		unsetsource();
 }
 
+void
+hsspread(Tokenrow *ntr, Tokenrow *trp, Nlist *np)
+{
+	Token *tp;
+	int hs;
+
+	hs = newhideset(trp->tp->hideset, np);
+	/* distribute hidesets */
+	for (tp=ntr->bp; tp<ntr->lp; tp++) {
+		if (tp->type!=NAME)
+			continue;
+		if (tp->hideset==0)
+			tp->hideset = hs;
+		else
+			tp->hideset = unionhideset(tp->hideset, hs);
+	}
+}
+
 /*
  * Expand the macro whose name is np, at token trp->tp, in the tokenrow.
  * Return trp->tp at the first token next to be expanded
@@ -186,17 +203,18 @@
 void
 expand(Tokenrow *trp, Nlist *np)
 {
+	static int depth;
 	Tokenrow ntr;
 	int ntokc, narg, i;
-	Token *tp;
 	Tokenrow *atr[NARG+1];
-	int hs;
 
+	depth++;
 	copytokenrow(&ntr, np->vp);		/* copy macro value */
 	if (np->ap==NULL) {			/* parameterless */
 		ntokc = 1;
 		/* substargs for handling # and ## */
 		atr[0] = nil;
+		hsspread(&ntr, trp, np);
 		substargs(np, &ntr, atr);
 	} else {
 		ntokc = gatherargs(trp, atr, (np->flag&ISVARMAC) ? rowlen(np->ap) : 0, &narg);
@@ -215,16 +233,9 @@
 			dofree(atr[i]->bp);
 			dofree(atr[i]);
 		}
+		hsspread(&ntr, trp, np);
 	}
-	hs = newhideset(trp->tp->hideset, np);
-	for (tp=ntr.bp; tp<ntr.lp; tp++) {	/* distribute hidesets */
-		if (tp->type==NAME) {
-			if (tp->hideset==0)
-				tp->hideset = hs;
-			else
-				tp->hideset = unionhideset(tp->hideset, hs);
-		}
-	}
+
 	ntr.tp = ntr.bp;
 	insertrow(trp, ntokc, &ntr);
 	trp->tp -= rowlen(&ntr);
@@ -343,6 +354,7 @@
 	}
 	return 0;
 }
+
 /*
  * substitute the argument list into the replacement string
  *  This would be simple except for ## and #
@@ -408,7 +420,7 @@
 				if(ttr.tp->hideset == 0)
 					ttr.tp->hideset = hs;
 				else
-					ttr.tp->hideset = unionhideset(ttr.tp->hideset, hs);
+					ttr.tp->hideset = unionhideset(rtr->tp->hideset, hs);
 				expandrow(&ttr, (char*)np->name);
 				for(tp = ttr.bp; tp != ttr.lp; tp++)
 					if(tp->type == COMMA)

Re: [9front] x509.c for ape in libsec.a

[kokamoto]

> Alright, I think I have a fix for this issue. We need to put the
> macro into the hideset before doing the final pass, or there's
> potential for macro expansion to cycle in some cases.
> 
> 	expand(A) => B
> 	expand(B) => A
> 	expand(A) => B

Thank you very much Ori.

Yes, it works fine here, too.

Kenji

Re: [9front] x509.c for ape in libsec.a

[ori]

>> Alright, I think I have a fix for this issue. We need to put the
>> macro into the hideset before doing the final pass, or there's
>> potential for macro expansion to cycle in some cases.
>> 
>> 	expand(A) => B
>> 	expand(B) => A
>> 	expand(A) => B
> 
> Thank you very much Ori.
> 
> Yes, it works fine here, too.
> 
> Kenji

It isn't quite right, though I haven't found any real-world
code that causes the problem, just artificial test cases.

I think this one should be regression-free.

diff -r 0cfcc2f8b898 sys/src/cmd/cpp/cpp.h
--- a/sys/src/cmd/cpp/cpp.h	Sat Jul 11 13:28:58 2020 -0700
+++ b/sys/src/cmd/cpp/cpp.h	Sun Jul 12 18:38:00 2020 -0700
@@ -110,7 +110,7 @@
 void	expand(Tokenrow *, Nlist *);
 void	builtin(Tokenrow *, int);
 int	gatherargs(Tokenrow *, Tokenrow **, int, int *);
-void	substargs(Nlist *, Tokenrow *, Tokenrow **);
+void	substargs(Nlist *, Tokenrow *, Tokenrow **, int);
 void	expandrow(Tokenrow *, char *);
 void	maketokenrow(int, Tokenrow *);
 Tokenrow *copytokenrow(Tokenrow *, Tokenrow *);
diff -r 0cfcc2f8b898 sys/src/cmd/cpp/macro.c
--- a/sys/src/cmd/cpp/macro.c	Sat Jul 11 13:28:58 2020 -0700
+++ b/sys/src/cmd/cpp/macro.c	Sun Jul 12 18:38:00 2020 -0700
@@ -169,9 +169,8 @@
 		}
 		if (np->flag&ISMAC)
 			builtin(trp, np->val);
-		else {
+		else
 			expand(trp, np);
-		}
 		tp = trp->tp;
 	}
 	if (flag)
@@ -186,18 +185,19 @@
 void
 expand(Tokenrow *trp, Nlist *np)
 {
+	static int depth;
+	int ntokc, narg, i, hs;
+	Tokenrow *atr[NARG+1];
 	Tokenrow ntr;
-	int ntokc, narg, i;
 	Token *tp;
-	Tokenrow *atr[NARG+1];
-	int hs;
 
+	depth++;
 	copytokenrow(&ntr, np->vp);		/* copy macro value */
 	if (np->ap==NULL) {			/* parameterless */
 		ntokc = 1;
 		/* substargs for handling # and ## */
 		atr[0] = nil;
-		substargs(np, &ntr, atr);
+		substargs(np, &ntr, atr, trp->tp->hideset);
 	} else {
 		ntokc = gatherargs(trp, atr, (np->flag&ISVARMAC) ? rowlen(np->ap) : 0, &narg);
 		if (narg<0) {			/* not actually a call (no '(') */
@@ -210,12 +210,13 @@
 			trp->tp += ntokc;
 			return;
 		}
-		substargs(np, &ntr, atr);	/* put args into replacement */
+		substargs(np, &ntr, atr, trp->tp->hideset);	/* put args into replacement */
 		for (i=0; i<narg; i++) {
 			dofree(atr[i]->bp);
 			dofree(atr[i]);
 		}
 	}
+
 	hs = newhideset(trp->tp->hideset, np);
 	for (tp=ntr.bp; tp<ntr.lp; tp++) {	/* distribute hidesets */
 		if (tp->type==NAME) {
@@ -343,19 +344,20 @@
 	}
 	return 0;
 }
+
 /*
  * substitute the argument list into the replacement string
  *  This would be simple except for ## and #
  */
 void
-substargs(Nlist *np, Tokenrow *rtr, Tokenrow **atr)
+substargs(Nlist *np, Tokenrow *rtr, Tokenrow **atr, int hideset)
 {
 	Tokenrow ttr, rp, rn;
 	Token *tp, *ap, *an, *pp, *pn;
 	int ntok, argno, hs;
 
 	for (rtr->tp=rtr->bp; rtr->tp<rtr->lp; ) {
-		if(rtr->tp->hideset && checkhideset(rtr->tp->hideset, np)) {
+		if(rtr->tp->hideset && checkhideset(hideset, np)) {
 			rtr->tp++;
 		} else if (rtr->tp->type==SHARP) {	/* string operator */
 			tp = rtr->tp;
@@ -405,10 +407,10 @@
 				*ttr.tp = *rtr->tp;
 
 				hs = newhideset(rtr->tp->hideset, np);
-				if(ttr.tp->hideset == 0)
+				if(hideset == 0)
 					ttr.tp->hideset = hs;
 				else
-					ttr.tp->hideset = unionhideset(ttr.tp->hideset, hs);
+					ttr.tp->hideset = unionhideset(hideset, hs);
 				expandrow(&ttr, (char*)np->name);
 				for(tp = ttr.bp; tp != ttr.lp; tp++)
 					if(tp->type == COMMA)