I confirmed that the new version has not broken anything in an intricate website that we just got working the other day, maersk.com. You can submit a parameter using their application and it returns information using XHR. So it exercises several different things: form submission, CSS issues, XHR, data attributes. Very cool, and it's all intact with the threading version. We were also wondering about speed recently. The XHR to retrieve container moves takes a moment, but it's not too bad. I can't tell if it got faster because of threading. b http://maersk.com 112782 4461 10 line after 148 has been deleted <> i1=MSKU0208730 <MSKU0208730> line 1 has been updated <TRACK> 11 <TRACK> i1* 11833 24 2 lines after 0 have been deleted 1 ? 1 lines 1 through 56 have been added <MSKU0208730> < error_icon> <SEARCH> < Print> From Yantian To Felixstowe
[-- Attachment #1: Type: text/plain, Size: 649 bytes --] The xhr system is necessarily sequential, as far as I can tell. Javascript rolls along and decides it needs some data, and asks for it by xhr, and stops, and waits, and really can't do anything else until that data is returned. There isn't any point to putting that download into a background thread, you still have to wait for it, so no, threading won't speed up any xhr requests. And this is almost always dynamic data so not cacheable. I guess a test might be a site with lots of js, like nasa, without any caching, and with or without jsbg. Even if it doesn't speed things up a lot I'm glad I did it, for several reasons. Karl Dahlke
> Even if it doesn't speed things up a lot I'm glad I did it, for several reasons.
I'm sure it will speed things up!
Does nasa have any database applications, deeper than the homepage? That
would be an interesting test of how good we are getting at multifaceted
problems.
Hi Karl, Thanks for the quick test - edbrowse www.nasa.gov - but unfortunately my build exits with an exception, in a thread, I think... it seems to be after the initial fetch from the url, which contain several script src=....js, along with other things... It seems to happen in http.c:808 - g->down_fd = creat(g->cdfn, 0666);, where g->down_force == 2... trying to track that down... note that the cdfn does contain some forward slashes - like "C:\\Users\\user\\Appdata\\Local\\Temp/edbrowse/0/back1524.1" - but I am not sure this is the problem... While in that first 'creat(... 1524.1)', the thread switches, and it is the next 'create(... 1524.2)' that triggers an exception dialog... very difficult to follow... So any discussion about maintaining a W32.x64 port is probably mute... until I can get it running... any ideas on that much appreciated... Hi Kevin, Thanks for the direct mail, and kind words, and I hope you do not mind if I answer here, since it is for a broader audience... Yes, to build edbrowse from source, in windows, the first thing to have is a suitable compiler... As has been mentioned there is cygwin... have tried it a few times... but not my choice... Another is MinGW-w64... I still use that now and then on some projects... to supply an alternate Win32 EXE... And there may be other non-native compilers... But to use the native Microsoft compiler, you must install MSVC... the 'Community' version is free... https://visualstudio.microsoft.com/vs/community/ ... You will also need the Windows SDK, but I think that is included in the install these days... So each of those choices involves a SUBSTANTIAL install, usually requiring admin rights, since in most cases they, write to the registry, permanently change the environment, particularly PATH... etc, etc, ... It may be possible, in some cases, to install each locally... that is for one user only, on a shared system... I certainly do that with MinGW... but this requires additional skill, to get it all functioning... But then you have to compile and install the considerable list of dependencies - duktape, libcurl, pcre, readline, pthreadVC2, zlib, ... what have I missed... It takes real effort download the sources for EACH of the above, compile each with the chosen compiler, and install them, where they can be found during the configuration stage of the cmake build of edbrowse... but I could help with that, if there was the interest... So compiling from source is a heavy effort... and suggests programming is already one of your (very time consuming) hobbies... An alternative is to build and publish binaries, like here http://www.edbrowse.org/static/ But again, that depends on the time and interest... Meantime, will try to get edbrowse running... there must be a way to avoid this exception... btw it also happens with 'b http://maersk.com'... in the same place... 2nd time thru... strange... Regards, Geoff.
[-- Attachment #1: Type: text/plain, Size: 628 bytes --] It sounds like the creat() call is not threadsafe. It opens a file on disk and while that io is running the other thread spins up and runs into the creat call to open a second file, and boom. But why would creat not be threadsafe? Do you have to specify a different, threadsafe version of the c library? You use to have to do this on unix long ago, but the threadsafe version has become the standard version. Oh and another complexity; if you do a pull you will find that creat call is gone, I do it a different way now, but still, that way worked on my side and no idea why it wouldn't work on windows. Karl Dahlke
Hi Karl, Not sure about create() and thread safety... since forever I have been adding a -D_REENTRANT flag to ALL builds, unix & w32... do I need to add/do anything else? ... have had no trouble using pthreads in other projects... did just recently update to ptw32 v.3.0.0... which required a minor CMakeLists.txt change, below... But anyway did a fetch and rebase to your latest... 1 minor clash... easily fixed... and the new EXE ran like hell was on fire... with b www.nasa.gov... lots of downloads, it seems... until it reached another exception... but different... In append0(), after many times thru, lots of threads created, Run-Time Check Failure #2 - Stack around the variable 'e' was corrupted. This is a stack char e[40], and on inspection after the exception, contains a string "l{a|000002CDF3254928,#document-fragment "... The fact that at that moment the buffer ends in a space, 0x20, and not a null, from sprintf(...) might indicate an overrun... inc'd to 80... got to native_insbf, with same e[40]... bumped to 80... then native_removeChild()... and I got to the end of the load... 3055 Testing done - = 191 ,p and got many lines, many of the form '* {...text...}'... some text like 'h3 Latest NASA Information on Hurricane Dorian... etc...' ... it looks good... What to test next? but out of time tonight... This is very good news, I think... will test more, and report soonest... Some other noted items - 1. In w32, using jsdb got an output like download in background not available on Windows at this time. Huh? Is that not what pthreads gives us... do not understand... 2. decided to build in ubuntu, but on trying to run ./edbrowse, got - .ebrc: SSL certificate file /etc/ssl/certs/ComSign_CA.pem does not exist or is not readable. edbrowse ready b www.nasa.gov the curl library returned the following error message while performing the requested operation: Problem with the SSL CA cert (path? access rights?) Any help with that appreciated... How can I add what is missing? 3. In unix build, found I had to add pthreads, under unix... diff --git a/CMakeLists.txt b/CMakeLists.txt index 5457acd..c0fac06 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -165,9 +165,14 @@ if (WIN32) message( FATAL_ERROR "*** Thread libray NOT FOUND!" ) endif() else () - message( FATAL "*** Threads NOT FOUNDunable to proceed") + message( FATAL_ERROR "*** Threads NOT FOUND! Unable to proceed") endif () endif () +if (UNIX) + find_package(Threads REQUIRED) + list(APPEND add_LIBS ${CMAKE_THREAD_LIBS_INIT}) + message(STATUS "*** list(APPEND add_LIBS ${CMAKE_THREAD_LIBS_INIT})" ) +endif () if (BUILD_EDBR_ODBC) # if the user OPTION ON # This module defines 4. Also a small Windows change for CMakeLists.txt, for pthreads update to 3.0.0... remove the kludge of defining HAVE_STRUCT_TIMESPEC, since the ptw32 now take care of that... And of course the question why such small fixed buffers like e[40], in jseng-duk.c. Anything wrong with making this 80, or something... maybe snprintf instead... Regards, Geoff.
> download in background not available on Windows at this time.
Oh ha ha.
That is a block because we used processes and windows didn't have fork, and so on and so on,
and I just forgot to remove the block.
Latest commit takes it away.
Karl Dahlke
The overrun bug you found has nothing to do with threading, and it's surprising nobody ran into it before. Latest commit should fix it. Still concerned that the system call creat() seemed not to be threadsafe. Yeah I'm not using it any more but still it should have worked. Maybe make a small threaded test program to create a few files in parallel and see what happens. Karl Dahlke
Hi Karl, Yes, the overrun has nothing to do with threading... I only ran into it running the special MSVC Debug build, which has a special prologue to write a marker, like oxcc, to ALL stack variable, and an epilogue that checks for invasions... of various types... very special case... And since in this case it seemed just by 1 byte, although there could have been others, which 'normally' will not cause a problem, so long as it is not killing a variable still in use, then the results are totally unpredictable... Maybe will not have time tonight to pull and test your latest... Ok, concerning 'creat()', I too think this system call would be 100% thread safe, why not? so I did what you suggested - found some simple code here - https://www.technical-recipes.com/2016/using-posix-threads-in-microsoft-visual-studio/ ... copied in your 'creat(buf, 0666)'... into the perform_work thread... and BANG got an exception... I was overlooking your permission value... 0666... the exception is an 'invalid parameter' dialog... when I replace that with '_S_IREAD | _S_IWRITE', 0x0180, everything ran smooth... 5 threads create, 5 temp files created... no problems... So this solves it in my mind... what about you? Will get to a pull soonest... Regards, Geoff.
Hi Geoff
Thanks for the remarks about Windows compilation. Well, I'll pull it up
this coming week and try MinGW and the community version of VS. At least
I'll try it!
> An alternative is to build and publish binaries, like here
Well if we did this, would we then get the potential promotional boost
Karl was mentioning of being able to say we run on all platforms?
Kevin
> Well if we did this, would we then get the potential promotional boost
You don't have to publish .exe files; you don't have to make it easy to build;
you don't have to have any windows users at all;
just *say* it runs on windows and there you go.
Actually a benefit of the windows port is it found a bug, and perhaps several bugs over the last few years,
that didn't come to light otherwise,
because different platforms expose different bugs.
Karl Dahlke
Hi Karl, We should certainly check the permission bits for 'open()'... the MS docs say - https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/open-wopen?view=vs-2019 It seems the perm.bits are ignored unless _O_CREAT is present, then it states "If a value other than some combination _S_IREAD and _S_IWRITE... generates an assert in Debug mode and invokes the invalid parameter handler..."... that seem clear enough... Now, in W32 stat.h, S_IREAD = 0x0100 and S_IWRITE = 0x0080. You have used open() some 18 times in the source, 10 of those have O_CREAT. It seems 2 of those use 0600, which is fine, = 0x180, but the other 8 use 0666, 0x0B6, which SEEMS bad... But running my 'b www.nasa.org' it passes thru say lock_fh = open(cachLock, O_WRONLY | O_EXCL | O_CREAT, 0666); many, many times, and MOST of the time it succeeds! And this is in Debug mode, so do NOT understand... Most example using 0666 are in fetchmail.c, about 5 times, and I have never tested that part of edbrowse... So do not know if we should be concerned... can you get by using just 0600? Open question... Hi Kevin, look forward to your trials to build in W32, from source... I do think there are some benefits in keeping a Windows build... and not just saying so... whether there are users or not... but always provided it does not take too much time and effort... and agree with Karl, the port has yielded some benefits in the past... Now that I have an edbrowse w32.x64 running, I can try to test my ftruncate/truncate DOSLIKE replacements, in my forks pthreads-w32 branch, which is where this started... will try to get to that... But can someone help me with my Ubuntu build? Here is what I see when running it - ~/projects/edbrowse/build$ ./edbrowse .ebrc: SSL certificate file /etc/ssl/certs/ComSign_CA.pem does not exist or is not readable. edbrowse ready b www.nasa.org 9194 the curl library returned the following error message while performing the requested operation: Problem with the SSL CA cert (path? access rights?) the curl library returned the following error message while performing the requested operation: Problem with the SSL CA cert (path? access rights?) the curl library returned the following error message while performing the requested operation: Problem with the SSL CA cert (path? access rights?) 1 lines 1 through 13 have been added = 14 ,p {Acheter ce domaine.} h1 nasa.org 2019 Copyright. All Rights Reserved. The Sponsored Listings displayed above are served automatically by a third party. Neither the service provider nor the domain owner maintain any relationship with the advertisers. In case of trademark issues please contact the domain owner directly (contact information can be found in whois). {Privacy Policy} qt ~/projects/edbrowse/build$ So something is wrong! How do I fix this? I have MANY things in my /etc/ssl/certs/ folder, but obviously NOT a ComSign_CA.pem that it seems to be looking for... Can I just alter the line in my ~/.ebrc - certfile = /etc/ssl/certs/ComSign_CA.pem to some other file/link... but which one? Each existing *.pem seems a link to elsewhere, like the most recent - lrwxrwxrwx 1 root root 61 Dec 23 2018 /etc/ssl/certs/TrustCor_RootCert_CA-2.pem -> /usr/share/ca-certificates/mozilla/TrustCor_RootCert_CA-2.crt And as mentioned, to even be able to compile edbrowse in ubuntu, I have to patch the CMakeLists.txt with - ```diff diff --git a/CMakeLists.txt b/CMakeLists.txt index 5457acd..c0fac06 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -165,9 +165,14 @@ if (WIN32) message( FATAL_ERROR "*** Thread libray NOT FOUND!" ) endif() else () - message( FATAL "*** Threads NOT FOUNDunable to proceed") + message( FATAL_ERROR "*** Threads NOT FOUND! Unable to proceed") endif () endif () +if (UNIX) + find_package(Threads REQUIRED) + list(APPEND add_LIBS ${CMAKE_THREAD_LIBS_INIT}) + message(STATUS "*** list(APPEND add_LIBS ${CMAKE_THREAD_LIBS_INIT})" ) +endif () if (BUILD_EDBR_ODBC) # if the user OPTION ON # This module defines ``` Surely any others using cmake MUST run into this problem? Hope is can be fixed soon... Regards, Geoff.
[-- Attachment #1: Type: text/plain, Size: 639 bytes --] Hi Geoff, There are several questions here, this only addresses one of them. The best bet is not to have certfile in your .ebrc config file at all. If you don't specify, curl looks for it and probably finds it, because it was configured that way. Only set certfile = if you need to override what curl does. If the certificates are not found, for any reason, then any https connection will fail. You can bypass this with the vs command. vs don't verify ssl connections. Then nasa and everything else will work - though if you're logging into an https site to do your banking it is less secure, obviously. Karl Dahlke
Hi Karl, Kevin, Re: .ebrc: certfile = /etc/ssl/certs/ComSign_CA.pem curl errors =============================================================== Just commented out that line, and everything worked fine... at least in my little tests... thanks... And I have had the SSL problems you sort of describe Kevin, particularly in the Windows build. When building curl, from source, you have to take care that it finds your build and install of SSL, plus OTHER things... or else you get a very broken library... but that's another topic... But generally, in linux, I usually depend that is is already installed, or is available thru the package manager, or apt, or ... and try to NEVER get into building from source, and installing, especially a system wide install... Trying $ curl https://weloveanimals.me looks like it runs fine... in both ubuntu, and windows... so I think I am good on the 'curl' front... thanks for the inputs... Re: CMakeLists.txt changes - 2 ============================== Yes, I have mentioned TWO (2), separate, CMakeLists.txt changes - one for Windows, and one for Unix... The change for Windows is only to get rid of a warning, since I upgraded my pthreads w32 installation... NOT important, and yes, could be included in a PR, if/when I get that far... But the change for Unix is critical to get the cmake build to work there... patch given in previous... This suggests other linux/unix people are `NOT` using 'cmake'! Or they would be shouting EEEEEK!... On reading the README, I can see your continue to support, and obviously use, of the src/makefile, or something... to me that is sad... maintaining TWO (2), or 3?, separate, build systems is always tricky... And the only way to maintain cmake, which is supposed to be for ALL systems, is to use it, play with modifying it, get to understand, and above ALL, trust that it will do the right thing... in ALL cases... make it the /ONLY/ build option... Karl, you are obviously maintaining the src/makefiles... and recently must have added '-lpthreads'... this has to be also done in CMakeLists.txt... just a few line of cmake script, as per my diff, and done... I await your decision on which way to jump on this ;=)) Re: ftruncate/truncate WIN32 replacements - w32 =============================================== For testing, I have used a compile time #ifdef DOSLIKE, but when, if, these w32 replacements work fine, then I would probably add something like a win32/wtrunc.c src file, like dirent.c and vsprtf.c... so the ugly #ifdef's can be removed from the main code... But at the same time, would like you to have a look at my forks, pthreads-w32 branch, and really ask is there not another way to do this... in a more portable way... Like not using ftruncate/truncate... maybe close and delete... or other ways... but maybe not... So I will try to get around to my very quick, kludgy, ugly, replacements... at least to try and determine for sure that equivalence has been achieved... soonest... Re: create(), open(), - possible invalid pmode parameters - w32 =============================================================== Will also try to get to the latest pull... and try retesting some of the open() changes... thanks for these... Re: sorry about mixed topics in 1 email ======================================= I can see you prefer separation... but I prepare all my replies in an editor, like gedit, offline, so to speak... so I like one file, like this ~/Documents/edbrowse/ebd-25-threads.txt, covering ALL the latest I have received, at the time of dispatch... I suppose I could try dispatching it topic by topic... but so could you split it up on receipt, separate on 'Re:', and reply to each separately... that is also ok by me... no probs... Regards, Geoff.
[-- Attachment #1: Type: text/plain, Size: 796 bytes --] > I can see you prefer separation... I have no preference; reply any way you wish; and I appreciate all your hard work. As per make - some of our distributers prefer using make. You can download virtually any package from any linux distribution, and the instructions tell you to type config, sometimes, and then make. I can't change how the world works and what the world wants. If I could, blind people everywhere would be using edbrowse, because, in my opinion, and it's just my opinion, pasting a screen reader on a graphical browser is like giving a newly blind pilot a 37,000 foot cane and telling him to go back to work. We all do the best we can. I'll have a look at truncate, see where and why it is used, see if there is an alternative. Please stand by. Karl Dahlke