[Edbrowse-dev] version

[Edbrowse-dev] version

[Karl Dahlke]

To recap, I created a nightmare with css parsing and querySelectorAll, because we had to, because some websites need it,
then I fixed the nightmare, so it doesn't bog down every site you look at.
With this in mind, and the other changes in the past few months, I think we should stabilize, then stamp a new version.
Fix bugs or make small enhancements (these are often the same thing), sure,
but no more major redesigns until 3.7.2.
Adam I still think we could put your gopher changes in, if you're up for that, I don't view that as a major redesign.
That would be a good thing to have.
But if you think that might not happen for a couple months then maybe move it to the next version.
What do you-all think?

Karl Dahlke

Re: [Edbrowse-dev] version

[Adam Thompson]

On Thu, Feb 15, 2018 at 06:56:35AM -0500, Karl Dahlke wrote:
> To recap, I created a nightmare with css parsing and querySelectorAll, because we had to, because some websites need it,
> then I fixed the nightmare, so it doesn't bog down every site you look at.
> With this in mind, and the other changes in the past few months, I think we should stabilize, then stamp a new version.
> Fix bugs or make small enhancements (these are often the same thing), sure,
> but no more major redesigns until 3.7.2.

Yeah sounds like a good idea.  To that end there're a couple of small fixes which we need:
- a form with an action of "" (or empty) is, as I recall, supposed to use the current url as its destination
(at least that's the behaviour I remember and on https://www.eaglemusicshop.com/ with the add to basket buttons)
- On that site there's also a button used as a link running the onclick code to send the browser to a new page.
That currently fails with the quite correct message that the button isn't part of a form.  I think we're
supposed to run the onclick code (or any click events) in any case.

> Adam I still think we could put your gopher changes in, if you're up for that, I don't view that as a major redesign.
> That would be a good thing to have.
> But if you think that might not happen for a couple months then maybe move it to the next version.

I think we set a time (subject to a bit of change) and if I've got something working then it goes in,
otherwise not.  We really need the current changes so they get priority.  If I miss with the gopher
stuff then that's a shame but not much more.

I've also removed a few unused things today as tidy-up.  I'm currently wondering where the mail server socket from
the mail sending mechanism is used as I can only find (with grep) a reference to it being defined and declared
as an extern but no actual usage.
I'm guessing it's actually used and don't use that part of the code so I'm not touching it.

Other than that, I'm not sure what else we need to sort, there may be a few more major changes we want,
but as previously stated, I think we need the css and query selector stuff quite urgently.

Cheers,
Adam.

[Edbrowse-dev] version

[Karl Dahlke]

> - a form with an action of "" (or empty) is, as I recall,
> supposed to use the current url as its destination

I fixed this.
I have notes that say this should not happen if the submit button has onclick code,
so that's how it rolls now.

> a button used as a link running the onclick code to
> send the browser to a new page.

This works for me, I just tested it, even going to a new page,
so not sure why it isn't working for you.

Karl Dahlke

Re: [Edbrowse-dev] version

[Adam Thompson]

On Thu, Feb 15, 2018 at 04:04:00PM -0500, Karl Dahlke wrote:
> > - a form with an action of "" (or empty) is, as I recall,
> > supposed to use the current url as its destination
> 
> I fixed this.
> I have notes that say this should not happen if the submit button has onclick code,
> so that's how it rolls now.

Thanks.

> > a button used as a link running the onclick code to
> > send the browser to a new page.
> 
> This works for me, I just tested it, even going to a new page,
> so not sure why it isn't working for you.

Thanks for the latest code which fixes this also.  I also noticed you removed the unused variable, thanks.
I wonder what else we should sort before making the release? Any thoughts?

[Edbrowse-dev] version

[Karl Dahlke]

[-- Attachment #1: Type: text/plain, Size: 211 bytes --]

I'm thinking maybe a week from tomorrow for version, unless of course something comes up, a bug, minor enhancement, I'm flexible, but sometimes you have to aim for a date or you don't get there.

Karl Dahlke

Re: [Edbrowse-dev] version

[Kevin Carhart]

Shall we do a code freeze on things like finding "slice"?  Just stockpile 
them til after the release.


> I'm thinking maybe a week from tomorrow for version, unless of course something comes up, a bug, minor enhancement, I'm flexible, but sometimes you have to aim for a date or you don't get there.
>
> Karl Dahlke
>

[Edbrowse-dev] version

[Karl Dahlke]

Things like URL.slice are exactly the kind of things I want to put in *before* a new version.
Two lines of code that might make some websites work that didn't work before, and really can't break anything.
I'm hoping you'll find more such fixes in the next week or so.

If you find a bug, or especially a seg fault, then definitely let me know.
I'm still somewhat uncomfortable about Chuck's bug, because it has not been diagnosed.
Might be curl's bug, probably is, but it might be mine.
I might have a workaround for it, but I'm still contemplating that one, and if so, should that be pre or post 3.7.2.
It's not unusual for me to work around bugs, e.g. a couple in tidy5.

Karl Dahlke

Re: [Edbrowse-dev] version

[Kevin Carhart]

> Things like URL.slice are exactly the kind of things I want to put in *before* a new version.

That's good to know, thank you.  I can probably find some more.

I wonder what would happen if.. I made a little script or ebrc that would 
load some number of the top websites.  Things like alexa.com have lists. 
Load each one at db3, log the whole thing and then grep for commonality, 
like if one TypeError appears 100 times out of 1000, we know we will get a 
lot of benefit for fixing it.

[Edbrowse-dev] version

[Karl Dahlke]

[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]

I don't know, it may be more complicated than that.
I use google.com a zillion times a day, and it may throw all sorts of js errors under edbrowse, but I don't care because I run with

nojs google.com

In other words, I can perform the public search without js.
facebook.com is probably a nightmare, but I can use mobile.facebook.com just fine, even without js.
The question might be, which sites are high runner, and important to function properly, and also require js.
These are usually more than just browse.
Some like amazon that you order products, or paypal.com for financial, or any of the banking sites.
These are the steepest in terms of js complexity, and they may even have those nasty js tests to see if your browser is proper,
but they are among the most important in terms of accessibility.
My entire line-oriented approach / philosophy doesn't really work if I can't do my banking online.
I'd like to never ever ever have to run a screen program, that would be my dream come true.

Here's another approach: any website that puts up a message
"Your browser doesn't support javascript", and I've seen a lot of these,
we should track through and find out why it is saying that.
It's like an acid test but real world.
And it won't be easy because a lot of those tests are in try catch and don't cause any errors that we can see.
Can we somehow see errors that are hidden inside try catch?
Maybe, see duktape.errCreate and Duktape.errThrow in the programmers guide.
And there are the acid tests themselves, which we should go back to some day.
Just a lot to do, and I need about 8 fulltime engineers to help me, but I haven't won the lotto yet.

Karl Dahlke

Re: [Edbrowse-dev] version

[Dominique Martinet]

Karl Dahlke wrote on Sun, Feb 18, 2018:
> And it won't be easy because a lot of those tests are in try catch and don't cause any errors that we can see.
> Can we somehow see errors that are hidden inside try catch?

I'd need to remove some of the dust but my duktape debugger test did log
errors inside try catch.
Definitely not something we want integrated before a release but could
help in a separate branch just to test, will send a mail with some
instructions once I got it to move again

-- 
Dominique

Re: [Edbrowse-dev] version

[Dominique Martinet]

Dominique Martinet wrote on Sun, Feb 18, 2018:
> Definitely not something we want integrated before a release but could
> help in a separate branch just to test, will send a mail with some
> instructions once I got it to move again


So, as a prerequisite, ou need a debugger-enabled duktape, or edbrowse
will segfault. There must be some way to detect if we can use it and
enable debugger only if this will work, but I don't know how yet.

To do that, you need to go to your duktape sources and edit
src/duk_config.h, look for DEBUGGER macros and change the undefs into
defines
I have enabled these:
#define DUK_USE_DEBUGGER_DUMPHEAP
#define DUK_USE_DEBUGGER_INSPECT
#define DUK_USE_DEBUGGER_PAUSE_UNCAUGHT
#define DUK_USE_DEBUGGER_SUPPORT
#define DUK_USE_DEBUGGER_THROW_NOTIFY

Debugger support is necessary, and requires DUK_USE_INTERRUPT_COUNTER
I think throw notify is what lets us print caught exceptions, we
wouldn't know otherwise.
Pause uncaught lets us print something on uncaught errors, we already do
that so you might not want it, but ultimately should let us run jdb
commands in the context of the error with local variables which might be
helpful.
I think inspect is what would let us run 'eval' commands within debugger
context, and dumpheap is a command to dump all local variables which I
haven't used yet either but both might be useful.

In duk_config.h, there also is DUK_USE_DEBUG which can print a lot of
messages from duktape, I tried this once but it does not help me much


Then make -f Makefile.sharedlibrary or build/install as usual

Then recompile edbrowse on my 'debugger' branch:
 git fetch https://github.com/martinetd/edbrowse.git debugger
 git merge FETCH_HEAD
(or whatever you usually do to pull changes)

And browsing should print caught throws, for example google.com gives me this:
./edbrowse http://google.com
13001
Enabling debugger
420
caught throw: TypeError: undefined not callable (property 'insertRow' of [object Object]) (line 244)
caught throw: TypeError: cannot read property 'getItem' of null (line 121)

(might want to change my prints to only show starting db3 or something,
but this is really prototype stage)


Hope this helps,
-- 
Dominique

Re: [Edbrowse-dev] version

[Kevin Carhart]

> I don't know, it may be more complicated than that.
> These are usually more than just browse.


True.  It does not solve the complex scenarios where you want to use a
website to do things, but I think it could be "necessary but not 
sufficient" some of the time, since a JS crash can inhibit dynamic 
behavior from working right.  Doing real ecommerce, or online banking, a 
lot of the time is going to be related to document.forms, submitting and 
maintaining state, right?  I've tried some of this.  It works some of the 
time and doesn't work some of the time.  And, although this could have 
many causes, I think one major problem is when a submission button isn't 
recognized, so you can't i*.

Anyway, I wanted to scratch an itch.. an interesting couple hours' 
research.  It is definitely not the whole story when we want to use the 
applications on sites and use them for what they're for.

Kevin

Re: [Edbrowse-dev] version

[Kevin Carhart]

This was a lot of good information in a short time.

Here's some discoveries.  Try ucla.edu, uiuc.edu and vmware.com .

I have a 5M file if you want it.  But you can make your own easily enough. 
I'm a fan of ebrc functions, by the way.  All I had to do was assemble 
some "b" statements in a function and start capturing.  They're great.





On Sat, 17 Feb 2018, Karl Dahlke wrote:

> Things like URL.slice are exactly the kind of things I want to put in *before* a new version.
> Two lines of code that might make some websites work that didn't work before, and really can't break anything.
> I'm hoping you'll find more such fixes in the next week or so.
>
> If you find a bug, or especially a seg fault, then definitely let me know.
> I'm still somewhat uncomfortable about Chuck's bug, because it has not been diagnosed.
> Might be curl's bug, probably is, but it might be mine.
> I might have a workaround for it, but I'm still contemplating that one, and if so, should that be pre or post 3.7.2.
> It's not unusual for me to work around bugs, e.g. a couple in tidy5.
>
> Karl Dahlke
> _______________________________________________
> Edbrowse-dev mailing list
> Edbrowse-dev@lists.the-brannons.com
> http://lists.the-brannons.com/mailman/listinfo/edbrowse-dev
>

--------
Kevin Carhart * 415 225 5306 * The Ten Ninety Nihilists

[Edbrowse-dev] version

[Karl Dahlke]

After a couple hours detective work, ucla.edu aborts because one of its javascript files is in utf16.
I should check for bom and convert utf16 or utf32 back to utf8.
How hard is this to do?
Don't know - fortunately edbrowse already contains the machinery.
Should it be done before 3.7.2?
Probably.
Should it also be done for css files?
Probably.

Karl Dahlke

[Edbrowse-dev] version

[Karl Dahlke]

[-- Attachment #1: Type: text/plain, Size: 217 bytes --]

js and css files now convert from whatever to utf8. As I say, the machinery was already part of edbrowse, so only a couple hours work.
Hope I wasn't too rushed or reckless.
www.ucla.edu now browses.

Karl Dahlke

[Edbrowse-dev] version

[Karl Dahlke]

Ok, so those three sites work now, but via more changes than I expected,
and hopefully said changes don't break something else.
It might be a little longer before I'm comfortable with a version, but that's fine.
Anything's better than a seg fault.

Karl Dahlke

[Edbrowse-dev] version / dyndns

[Kevin Carhart]

Thank you for working on them!  I noticed that you wrote a 
searchscripts().  That's useful.  I was doing the same thing recently, 
crudely with blah.split("mystring").length

But that is an example of where long expressions in jdb, with a varying 
bit in the middle of a long static bit, is annoying to continually 
assemble over and over, so searchscripts is easier, and easy to memorize.

And static analysis is one way of tackling errors in event code, where we 
don't have a filename or line number reported, right?

I'm also replying here to Chuck's report about dyndns.  Thanks for raising 
this, Chuck.

Well, I noticed this when I run dyndns.com with db3:

TypeError: undefined not callable (property 'getClientRects' of [object 
Object])
failure on 0xf63938.onload()
execution complete

I'm going to investigate if we can satisfy this with some stubs.  Maybe it 
will be similar to our set of stubs for Canvas.  There's a 
related function, getBoundingClientRect.  Both of these things had 
runtime errors in quite a few sites in my top sites experiment yesterday. 
So by fixing it for dyndns we should also fix it for some other 
situations at the same time.

https://developer.mozilla.org/en-US/docs/Web/API/Element/getClientRects
https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect

Kevin

[Edbrowse-dev] update on dyndns

[Kevin Carhart]

Actually, the rectangles error happens later on, so I don't think it is 
related to this particular unresponsive button.  Though it is still 
probably useful overall to supply the rectangles thing later.
But there is a bad JSON error in mmapi.js which happens adjacent to the 
request for "notice", which brings back and processes the div that contains these 
buttons.  I bet that the code that is supposed to imbue these buttons does 
not complete because of the JSON problem.  I'm in mmapi a little, and I 
think it has to do with getting and setting cookies.  "Storage" is 
mentioned, but it does not appear to be hitting our localStorage.  The 
functions are "get" and "set", which is not what localStorage uses.

The bad JSON seems to be a name with no value, like {"v":}



  On Sun, 18 Feb 2018, Kevin Carhart wrote:

>
> Thank you for working on them!  I noticed that you wrote a searchscripts(). 
> That's useful.  I was doing the same thing recently, crudely with 
> blah.split("mystring").length
>
> But that is an example of where long expressions in jdb, with a varying bit 
> in the middle of a long static bit, is annoying to continually assemble over 
> and over, so searchscripts is easier, and easy to memorize.
>
> And static analysis is one way of tackling errors in event code, where we 
> don't have a filename or line number reported, right?
>
> I'm also replying here to Chuck's report about dyndns.  Thanks for raising 
> this, Chuck.
>
> Well, I noticed this when I run dyndns.com with db3:
>
> TypeError: undefined not callable (property 'getClientRects' of [object 
> Object])
> failure on 0xf63938.onload()
> execution complete
>
> I'm going to investigate if we can satisfy this with some stubs.  Maybe it 
> will be similar to our set of stubs for Canvas.  There's a related function, 
> getBoundingClientRect.  Both of these things had runtime errors in quite a 
> few sites in my top sites experiment yesterday. So by fixing it for dyndns we 
> should also fix it for some other situations at the same time.
>
> https://developer.mozilla.org/en-US/docs/Web/API/Element/getClientRects
> https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect
>
> Kevin
>
> _______________________________________________
> Edbrowse-dev mailing list
> Edbrowse-dev@lists.the-brannons.com
> http://lists.the-brannons.com/mailman/listinfo/edbrowse-dev
>

--------
Kevin Carhart * 415 225 5306 * The Ten Ninety Nihilists