[Edbrowse-dev] xqsuperschool.org (fwd)

[Edbrowse-dev] xqsuperschool.org (fwd)

[Kevin Carhart]

Karl sez:
> This also browses to an empty page, like NASA use to do.
> Add to the list of troublesome sites.


Thanks, I will add xqsuperschool.org.  I actually have a list.

There is a juicy problem I would like to raise with youall, because there 
may be an easy way to implement some of it.

Now that I have done find & fix for a while, patterns are emerging for how 
to do detective work in what is wrong with a particular site.  I'd like to 
mention a couple of these in case there is something that would not be 
that difficult to implement which would improve our ability to find and 
isolate a particular missing piece of DOM, or whatever it may be.  Once 
you finally have that line of JS that says style.removeAttribute(filter), 
you can say "it expects style to have attributes."  And you can add those 
and get it into edbrowse.  But finding that line is a problem.

(A) So first of all, do we still have that ability to make edbrowse 
terminate on the first runtime error, at a high db level?  Could this be 
brought back on an obscure high number like db9, where it won't bother 
anyone?  Or on a flag you can enable/disable as we can do with js?  I 
observed that doing an edbrowse run has similarities to running gcc on 
source code with errors.  gcc may report three pages' worth of errors, 
but it is the first error you most want to know about.  Everything after 
that may be the cascading effects of the messed-up state caused by not 
being able to resolve whatever error #1 was about.  So that's why this 
trick was invaluable in edbrowse.

(B) Here's a big one.  Do we know in a clear way, what is the reason why 
some JS runtimes report line numbers and some do not?  This has drastic 
effects on trying to work.  I believe that minified code is part of the 
problem.  The line numbers may only go from 1-3.  I found it was actually 
effective to use esprima.parse and escodegen.generate in rapid 
succession.  esprima.parse converts javascript into an AST (Abstract 
Syntax Tree) and escodegen.generate turns an AST back into javascript with 
proper linefeeds.  Don't worry - I am not trying to introduce anything 
about ASTs themselves.  The point is that if you run these two things in 
sequence, you end up with runnable JS, only now it will report "TypeError 
in line 3456" rather than "TypeError in line 1".

So this is good, but there is currently manual work involved, so maybe 
there is an easier way to accomplish this.  Here's a sample situation of 
how this may happen:

(1) I load xqsuperschool.org to try to work on it.
(2) The site reports a javascript runtime error in jquery.min.js, line 2
(3) I quit edbrowse
(4) I go get my esprima/escodegen code, append it on startwindow and 
recompile
(5) I load xqsuperschool again
(6) I go to jdb and echo different parts of document.scripts.
document.scripts[0].data is small
document.scripts[1].data is small
document.scripts[2].data is massive and I can tell from echoing it that it 
is minified.

So after some disambiguation, I think document.scripts[2].data is my 
jquery.

(7) x1 = esprima.parse(document.scripts[2].data)
(8) x2 = escodegen.generate(x1)
(9) I use my terminal program and the clipboard to capture the de-minified 
jquery
(10) In my local /edbrowse/src folder, I start a new file, paste in the 
de-minified jquery and save it.
(11) I don't have any way of making the live xqschools.org substitute my 
de-minified jquery.  So I wget xqschools.org (to 
/edbrowse/src/xq.html)  I'm going to work entirely 
local.  The benefit is that I can point the home page towards de-minified 
js files.  The drawback is that I could introduce confounds.  You're now 
working on a local replica of the user's experience in getting 
xqschools.org from the internet, not the real thing.
(12) My xq.html has several references to js files 
and css files.  I also wget all of these to my /edbrowse/src
(13) I open xq.html in an editor and change the js and css paths one by 
one.  Now I hope I have a representative replica, though I don't know that.
(14) Now I browse xq.html in edbrowse.  Indeed, it often works, and now it 
crashes on my error in jquery, on line 3456 rather than line 2.
(15) I can finally work!  I open jquery, go to 3456 and find out that we 
are missing childNodes.item or style.attributes or something.
(16) Add this to startwindow, try it and find out whether it now breaks on 
something new or doesn't break at all.
(17) Write a patch about the missing DOM thing

(C) If this isn't especially easy or possible on our end, I'll write and 
ask Sami.  But for 
this new ability to put a function body on .body, can this be extended 
easily?  If I create a function entirely within js, the body 
isn't accessible:

xxx = function() { alert('hi') }
function () { [ecmascript code] }
xxx.body
undefined

Is there any easy way to extend .body to this?  And thank you for 
implementing .body in the first place.

(D) Final thing.  As I read the output from db3 or higher, the "execute" 
lines are invaluable.  That basically gives you a context of where the bug 
is occurring.  So I think as of a few days ago we now have tons more event 
handlers running successfully.  So take nasa.gov, for instance.  There is 
a long sequence towards the end:

execute onclick
execution complete
execute onclick
execution complete
execute onclick
execution complete

Now let's say that on nasa or something else, there's a runtime error in 
one of those handler runs:

execute onclick
could not run blah on undefined

This time I can't apply esprima and escodegen.  I don't have my bearings 
at all for what this is.  Is there any way to flesh out the calling 
context?  What code is failing?  What element is it on?  Maybe this is a 
Sami question?

I need to fire this email immediately because it's onlyin memory.. if my 
connection went down right now I would be in a world of hurt.  Thanks for 
reading.  If we can address any of these it will pay off by a lot in being 
able to isolate problems more efficiently.

Kevin
































---------- Forwarded message ----------
Date: Fri, 8 Sep 2017 06:44:43
From: Karl Dahlke <eklhad@comcast.net>
To: kevin@carhart.net
Subject: xqsuperschool.org

This also browses to an empty page, like NASA use to do.
Add to the list of troublesome sites.

Karl Dahlke

[Edbrowse-dev] xqsuperschool.org (fwd)

[Karl Dahlke]

[-- Attachment #1: Type: text/plain, Size: 1141 bytes --]

You raise 20 questions; here is just the first thought.
Many times I need to work locally to debug, as you point out, but don't forget about the <base> tag.
Put <base href=original-url> at the start of the file and then you don't have to have localized every css every js etc.
Just the one you're modifying or unminimizing.

We have jdb and many other debugging facilities built into edbrowse,
and I have no problem with that as long as the user doesn't accidentally stumble upon them.
More flags, more options to do more things, I have no problem with that.
You have done more in this area than I, so think of how we might activate some of these automatically based on a flag or some such.
Deminimizing js as we go is probably the most valuable, maybe a jmin toggle flag or something,
but as you say you have to dip into scripts[n].data, since the original source is mminimized.
Well scripts[n].src tells you wherre the js came from, so probably not hard to put it all together.
I assume your deminimizing code is open source and could be put into startwindow.js with the proper license, as we did with jotform.

Karl Dahlke

Re: [Edbrowse-dev] xqsuperschool.org (fwd)

[Kevin Carhart]

On Fri, 8 Sep 2017, Karl Dahlke wrote:

> You raise 20 questions; here is just the first thought.

I know, sorry :)  Well anyway, writing it to edbrowse-dev also doubles as 
documenting the situation as of now, which I & others can refer back to.

> Put <base href=original-url> at the start of the file and then you don't have to have localized every css every js etc.

Thanks for this.  It will save a step with the local replica technique.

> I assume your deminimizing code is open source and could be put into 
> startwindow.js with the proper license, as we did with jotform.

It is open source.  I have it here, in the file called es.js:

https://github.com/KevinCarhart/esbrowse.git

The original projects are:
https://github.com/jquery/esprima.git
https://github.com/estools/escodegen.git

If there is no objection to making the user's download burden slightly 
bigger for something that is mostly for debugging, it is very useful for 
that purpose.  Come to think of it, if the users did have this, they might 
more often be able to report an error message back to us with a line 
number in it.

K

[Edbrowse-dev] xqsuperschool.org (fwd)

[Karl Dahlke]

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

> Come to think of it, if the users did have this, they might
> be able to report an error message back to us with a line number in it.

That is true iff the feature runs all the time, and not based on some toggle flag.
It's hard to imagine that impacting performance significantly.
It's just one more step in js compilation.
But ...

1. If the code is clean, not minimized, pushing it through this machine could make it worse, not better.
In fact we might skip this step for any js with more than 20 lines, as that is probably not minimized.
So if jmin is set and line count < 20 then deminimize it.

2. We would have to know this machine works for *every* js out there.
A utf8 char or a really weird regular expression won't derail it.
So it's another possible point of failure.

So I might like to keep this feature for us, based on a flag or something, and then the normal user
wouldn't give us a meaningful line number, but we probably have to reproduce his bug in our world anyways, so we just turn the flag on.

Karl Dahlke

Re: [Edbrowse-dev] xqsuperschool.org (fwd)

[Kevin Carhart]

> So I might like to keep this feature for us, based on a flag or 
> something, and then the normal user

That would be great from my perspective.  I set things like db3, ua1 
before loading a website as it is.  Or I could put it in my ebrc if it was 
too many steps.

> wouldn't give us a meaningful line number, but we probably have to 
> reproduce his bug in our world anyways, so we just turn the flag on.

True.  This would be just as good.