edbrowse-dev - development list for edbrowse
 help / color / mirror / Atom feed
* [edbrowse-dev] Threads
@ 2019-08-27  7:19 Karl Dahlke
  2019-08-28  1:30 ` Geoff McLane
  0 siblings, 1 reply; 11+ messages in thread
From: Karl Dahlke @ 2019-08-27  7:19 UTC (permalink / raw)
  To: edbrowse-dev

[-- Attachment #1: Type: text/plain, Size: 1987 bytes --]

Folks, it has happened; i'm using threads for asynchronous downloads instead of processes.
The word fork is not to be found in the edbrowse source!

There are two places this occurs.

1. Download files in background.
This is not the default.
Use the bg command to turn this on.
Usually when a file is large and you just want to let it run in the background.
Type bglist for a list of the background jobs.
Careful, it's not a separate process, so if you quit edbrowse before the download is complete, it just stops and doesn't finish.

2. Download all the javascript files in the background and in parallel.
These can be large files so I'm hoping this speeds things up.
This is on by default because you usually want it, besides,
I really need folks to test it under various conditions
and find any weird race conditions or thread unsafe code or seg falts etc.
It works with curl+gnutls, which the process implementation never did.

Both of these have lots of corner cases, especially #1.
Download ftp files, gopher files, https files, several at onces, some large,
does it all work? Do they interact with each other?
Does the bglist command work?
Does it show the sizes of the downloads in progress?
Does a foreground download still give you the dots?

Are javascript files pulled from cache or put into cache, even when they download in background?
(This is important.)

Then there is a whole host of followup work, like putting off the async scripts til the end,
and maybe starting the js fetches even earlier at parse time,
and maybe fetching css files in background (which isn't done yet),
and on and on.
But first let's make sure I didn't break all sorts of things with this commit.

Set db3 and you'll see all the fetches and http codes etc in a jumbled order as the files are loaded.
Type jsbg to go back to a sserial order.

I've never touched posix threads before so give me a pat on the back for diving into something new.

Karl Dahlke

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [edbrowse-dev] Threads
  2019-08-27  7:19 [edbrowse-dev] Threads Karl Dahlke
@ 2019-08-28  1:30 ` Geoff McLane
  2019-08-28  1:48   ` Karl Dahlke
  0 siblings, 1 reply; 11+ messages in thread
From: Geoff McLane @ 2019-08-28  1:30 UTC (permalink / raw)
  To: Karl Dahlke, edbrowse-dev

Hi Karl,

This use of pthread.h opens the w32 port... I forget since when,
the cmake build of the windows port **required** this... maybe
since the beginning...

That means I was able to quickly code, an UNTESTED port, using
the DOSLIKE flag...

So in my fork, I have coded an **untested** set of fixes,
mainly on ftruncate/truncate WIN32 replacements...

See https://github.com/geoffmcl/edbrowse/tree/pthreads-w32

I have not yet created a PR, since I would like to test,
and confirm, the replacements work equivalently, and then
remove my very loud TODO flags... which signal untested...

But what is the use? Simply are there any edbrowse users
running in Windows? Who care about, uses, a W32 port...

If not why not? Need to begin to understand...

Is setting up the edb dependencies just too difficult? Can
we help with that??

Or what??? Other ways to read/navigate the web... big question!

I could do some testing, if given a sort of baby script to
read a web site, where multiple js and css are needed,
before rendering the page... trap, and check equivalence...

Certainly need big HELP testing...

And you certainly need a pat on the back for diving into
something new...

Regards, Geoff.


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [edbrowse-dev] Threads
  2019-08-28  1:30 ` Geoff McLane
@ 2019-08-28  1:48   ` Karl Dahlke
  2019-08-28  3:51     ` Kevin Carhart
  0 siblings, 1 reply; 11+ messages in thread
From: Karl Dahlke @ 2019-08-28  1:48 UTC (permalink / raw)
  To: edbrowse-dev

[-- Attachment #1: Type: text/plain, Size: 1566 bytes --]

There never has been an edbrowse windows user and there may never be.
As others have pointed out on this list, edbrowse dovetails with other command line tools in a unix or even mac environment,
and isn't nearly as useful on windows.
In other words, it is sometimes edbrowse plus other things that provides the power.
So why did we bother one might ask?

Well, at times I had dreams of getting grants, actually getting paid to work on edbrowse, and maybe subcontracting to some of you as well,
and grants are easier to get if you can claim your software runs on every platform.
Believe me the whole blind industry is windows centric, jaws, nvda, window eyes, on and on.
Rehab and employment centers teach windows and only windows.
So it was mostly for marketing.

But it turns out I can no more get a grant than I can swim the ocean.
It's just not one of my skills.
So if we give up on that, then I really don't worry about the windows port much any more, unless it is easy to maintain.
I've learned a lot though from it, and I thank you for all your hard work,
plus your continued tidy support.

Certainly switching to threads and away from fork and processes and wait and so on makes it easier.

An easy test is

edbrowse www.nasa.gov

That exercises all sorts of stuff, including a dozen javascript files which are fetched and loaded in parallel by threads.
If you get about 170 lines of stuff

=
170
,p

then it's all good.

You can watch the fetches in parallel and other things at debug level 3

db3
ub
b


Karl Dahlke

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [edbrowse-dev] Threads
  2019-08-28  1:48   ` Karl Dahlke
@ 2019-08-28  3:51     ` Kevin Carhart
  2019-08-28  4:16       ` Karl Dahlke
  0 siblings, 1 reply; 11+ messages in thread
From: Kevin Carhart @ 2019-08-28  3:51 UTC (permalink / raw)
  To: Edbrowse-dev


Congratulations on the great work!  And commendations to Geoff also.  I 
would be a candidate to compile for Windows but I think the expense 
of the commercial compilers is probably what has stopped me when I 
have occasionally thought about doing it.  It says "visual C studio is 
assumed on Windows."  I never knew what to do when I read this.  I'm 
sorry to ask this in a sleepy way after there was probably a thread about 
it a while ago, but is there any chance of (a) cygwin or (b) DOS/CMD? 
These environments are both a natural analog for CLI unix, are they not?

I was just telling Geoff offlist that edbrowse would be absolutely lovely 
and useful at my new job, so this could be the moment when I become the 
guinea pig (and even work on it during the day a little?  :)  ) and it 
will redound to the firm's benefit, I'm sure.

Either way, thank you Geoff, Jim and others.  Tidy5 developers should be 
commended early and often, it's such a crucial backbone I hardly think 
about it anymore!




  On Tue, 27 Aug 2019, Karl Dahlke wrote:

> There never has been an edbrowse windows user and there may never be.
> As others have pointed out on this list, edbrowse dovetails with other command line tools in a unix or even mac environment,
> and isn't nearly as useful on windows.
> In other words, it is sometimes edbrowse plus other things that provides the power.
> So why did we bother one might ask?
>
> Well, at times I had dreams of getting grants, actually getting paid to work on edbrowse, and maybe subcontracting to some of you as well,
> and grants are easier to get if you can claim your software runs on every platform.
> Believe me the whole blind industry is windows centric, jaws, nvda, window eyes, on and on.
> Rehab and employment centers teach windows and only windows.
> So it was mostly for marketing.
>
> But it turns out I can no more get a grant than I can swim the ocean.
> It's just not one of my skills.
> So if we give up on that, then I really don't worry about the windows port much any more, unless it is easy to maintain.
> I've learned a lot though from it, and I thank you for all your hard work,
> plus your continued tidy support.
>
> Certainly switching to threads and away from fork and processes and wait and so on makes it easier.
>
> An easy test is
>
> edbrowse www.nasa.gov
>
> That exercises all sorts of stuff, including a dozen javascript files which are fetched and loaded in parallel by threads.
> If you get about 170 lines of stuff
>
> =
> 170
> ,p
>
> then it's all good.
>
> You can watch the fetches in parallel and other things at debug level 3
>
> db3
> ub
> b
>
>
> Karl Dahlke
>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [edbrowse-dev] Threads
  2019-08-28  3:51     ` Kevin Carhart
@ 2019-08-28  4:16       ` Karl Dahlke
  0 siblings, 0 replies; 11+ messages in thread
From: Karl Dahlke @ 2019-08-28  4:16 UTC (permalink / raw)
  To: Edbrowse-dev

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

It works under cygwin, but that isn't really a windows port it's a unix emulator under windows.
If that meets your needs then all good.

As for the compiler I imagine any compiler would do, and there ought to be a free one for windows somewhere.
If not, that's just one more strike against windows, along with the up front cost of buying the damn os in the first place.

Karl Dahlke

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Edbrowse-dev] Threads
  2014-12-05 21:16 [Edbrowse-dev] Threads Karl Dahlke
@ 2014-12-05 22:05 ` Adam Thompson
  0 siblings, 0 replies; 11+ messages in thread
From: Adam Thompson @ 2014-12-05 22:05 UTC (permalink / raw)
  To: Karl Dahlke; +Cc: Edbrowse-dev

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

On Fri, Dec 05, 2014 at 04:16:36PM -0500, Karl Dahlke wrote:
> I think a seg fault in any thread brings down the entire process -
> so perhaps leaning a little towards separate processes,
> if we do separate functionality.

Indeed, that was my thinking. Although threads are lighter in terms of overhead, they also don't provide the isolation and flexibility we need.

> Can still use ipc, in its various forms, for communication,
> even common memory for global or session variables,
> though I don't know how well ipc ports to windows.

I'm not sure about portability either (see earlier emails)
but for the minute I've done some work with this stuff on Linux before and have
a few ideas how we could put things together.

> Sure, processes are less efficient than threds,
> but we're not doing a number crunching simulation here.

Yeah, and as stated above, we get more flexibility with processes.

Cheers,
Adam.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Edbrowse-dev]  Threads
@ 2014-12-05 21:16 Karl Dahlke
  2014-12-05 22:05 ` Adam Thompson
  0 siblings, 1 reply; 11+ messages in thread
From: Karl Dahlke @ 2014-12-05 21:16 UTC (permalink / raw)
  To: Edbrowse-dev

I think a seg fault in any thread brings down the entire process -
so perhaps leaning a little towards separate processes,
if we do separate functionality.
Can still use ipc, in its various forms, for communication,
even common memory for global or session variables,
though I don't know how well ipc ports to windows.
Sure, processes are less efficient than threds,
but we're not doing a number crunching simulation here.

Karl Dahlke

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Edbrowse-dev] Threads
  2014-12-05 19:29 Karl Dahlke
@ 2014-12-05 20:32 ` Adam Thompson
  0 siblings, 0 replies; 11+ messages in thread
From: Adam Thompson @ 2014-12-05 20:32 UTC (permalink / raw)
  To: Karl Dahlke; +Cc: Edbrowse-dev

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

On Fri, Dec 05, 2014 at 02:29:11PM -0500, Karl Dahlke wrote:
> Honestly, it sounds like it adds a lot of work to a project that is already
> a lot of work, and there's only three of us.
> I think the goal is to have js not seg fault and not infinite loop,
> and then there's no issue.

Ultimately that'd be brilliant, but js is a powerful language,
and without having the engine execute in its own process or thread,
there's no easy way to monitor it since there are just so many ways of causing
js-related problems.

In addition, I like the idea of being able to switch buffers and get on with
work whilst a page loads in another buffer,
and it gives us a better mechanism of implementing js stuff which is genuinely
time-critical (i.e. there are some pages I've used where ajax is used and a
synchronous version just won't work due to values such as session keys timing 
out).

It's going to be a lot of work to get right,
but modern browsers are complex things,
and edbrowse (with the extra functionality of a text editor and email client) is more so.
However, if we want it to remain relevant,
we'll need to handle this complexity some day or else fall further and further 
behind.

Cheers,
Adam.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Edbrowse-dev] Threads
@ 2014-12-05 19:29 Karl Dahlke
  2014-12-05 20:32 ` Adam Thompson
  0 siblings, 1 reply; 11+ messages in thread
From: Karl Dahlke @ 2014-12-05 19:29 UTC (permalink / raw)
  To: Edbrowse-dev

Honestly, it sounds like it adds a lot of work to a project that is already
a lot of work, and there's only three of us.
I think the goal is to have js not seg fault and not infinite loop,
and then there's no issue.
That is probably something we can achiev,
perhaps more easily with an engine that is less persnickety
and more intuitive than mozilla.
I recommend making the switch first, and if we still have issues,
we can change the function calls to ipc calls later, if need be,
but I don't think we're going to have these kinds of issues.
Hope not anyways.

Karl Dahlke

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Edbrowse-dev] threads
  2014-03-21 22:56 [Edbrowse-dev] threads Karl Dahlke
@ 2014-03-22 21:30 ` Adam Thompson
  0 siblings, 0 replies; 11+ messages in thread
From: Adam Thompson @ 2014-03-22 21:30 UTC (permalink / raw)
  To: Karl Dahlke; +Cc: Edbrowse-dev

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

On Fri, Mar 21, 2014 at 06:56:33PM -0400, Karl Dahlke wrote:
> To maintain the thread on an email list,
> save the mail you want to reply to unformatted,
> edit the email, browse, type re or rea,
> note the second line, it's a reference line,
> this is what is needed to maintain the thread.

That's interesting, I've always wondered how mutt seems to do this automagically.

Cheers,
Adam.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Edbrowse-dev] threads
@ 2014-03-21 22:56 Karl Dahlke
  2014-03-22 21:30 ` Adam Thompson
  0 siblings, 1 reply; 11+ messages in thread
From: Karl Dahlke @ 2014-03-21 22:56 UTC (permalink / raw)
  To: Edbrowse-dev

To maintain the thread on an email list,
save the mail you want to reply to unformatted,
edit the email, browse, type re or rea,
note the second line, it's a reference line,
this is what is needed to maintain the thread.

Karl Dahlke

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2019-08-28  4:16 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-27  7:19 [edbrowse-dev] Threads Karl Dahlke
2019-08-28  1:30 ` Geoff McLane
2019-08-28  1:48   ` Karl Dahlke
2019-08-28  3:51     ` Kevin Carhart
2019-08-28  4:16       ` Karl Dahlke
  -- strict thread matches above, loose matches on Subject: below --
2014-12-05 21:16 [Edbrowse-dev] Threads Karl Dahlke
2014-12-05 22:05 ` Adam Thompson
2014-12-05 19:29 Karl Dahlke
2014-12-05 20:32 ` Adam Thompson
2014-03-21 22:56 [Edbrowse-dev] threads Karl Dahlke
2014-03-22 21:30 ` Adam Thompson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).