Re: [Edbrowse-dev] Messages to and from edbrowse-curl

[Adam Thompson <arthompson1990@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6107 bytes --]

On Fri, Jan 01, 2016 at 10:40:35AM -0500, Karl Dahlke wrote:
> > I'm thinking probably that each user would need their own edbrowse-curl,
> 
> If I, as a single user, could still have multiple instances
> of edbrowse connecting to it, from different consoles,
> accessing similar websites and sharing one cookie space,
> then that makes sense and is fine.
> /tmp/edbrowse directory could contain a file per user indicating
> the process of edbrowse-curl running, if such is running,
> /tmp/edbrowse/run-curl-user,
> or spawn one otherwise, similar to the /var/run files in linux
> though I don't think there is such a standard in windows
> so we may as well use /tmp/edbrowse for this purpose on both OSs,
> again keeping the code as much the same in both worlds.
> This directory already exists and is used for temp files for plugins,
> e.g. converting pdf to html in preparation for browse, etc etc.
> Could also be used for temp files if we choose to use that mechanism
> to pass the http data back to edbrowse.

Agreed, though we could just as well use the correct directories on Linux and
Windows if we're hiding all of this stuff in functions anyway.

> > No, I think we just download the data,
> > may be have a status message and then it's up to edbrowse what it does.
> 
> It could be  a personal preference, but I don't want it to work that way.
> I don't want to sit there not knowing, and then later the download is done,
> I want to see the progress dots as they happen, in real time,
> otherwise why have them. I mean that's the whole point.

Yeah, it depends on how you use it. To clarify what I'm saying,
I'm proposing that edbrowse-curl supports a status message,
and then it is up to edbrowse whether it checks the status of the download and
prints dots, checks the status of the download on command or whatever.
Basically though, edbrowse-curl doesn't fire dots at edbrowse in realtime
because that's a UI thing first and foremost.

> I also like to see the debug messages as they happen, in real time,
> not spewing out at the end after download.
> More than once edbrowse has been in an infinite loop,
> despite our loop counter in http.c, an infinite redirection loop,
> and I see that with db2, see the websites it is vectoring through,
> or maybe the same site again and again, and I hit control c to stop it.
> Anyways that's just an example - I want to see things as they happen.
> It really isn't hard to do.
But it is bad design. What I'm thinking we should do here is again have
edbrowse-curl produce all this stuff (either as messages or on command,
may be using a polling loop of some kind)
and then leave it up to edbrowse what it does, i.e.
we may want to put them in a log etc.
Actually, I'd quite like a way to enable debug logging to a file from within an
edbrowse session without having to quit and start script etc.

> > > - Looks like a file you'll want to download, supply a file name or x to abort
> > Again, I'd just download the data, if we want to pull it into memory then we
> > can potentially do that, but mean while the data download carries on.
> 
> Small files just don't matter, and for big downloads you've
> lost your flexibility here.
> Maybe you don't have enough ram for that big iso image,
> and maybe /tmp/edbrowse, or wherever you put it by default,
> doesn't have room either.
> It's on a basic windows drive like c: and you really have room on d:,
> so curl asks you where you want to download it as soon as it gets the headers
> and you direct it somewhere else.
> It's a little more work to do it this way,
> but not much more really, a few messages,
> and that's how it works now,
> and it will let some people do some things on some computers
> that could not be done with an autodownload into a predefined place
> and then figure it out later philosophy.
Yeah that makes sense actually, but what do we do with the download in the 
meantime?
I'm not sure if we should just pause waiting or not.
Perhaps start writing to a temp file,
then when the user selects a file or whatever dump what we've downloaded so far
to the new location? I'm just not sure what servers'll do if we stop reading
from them for a while, probably drop the connection.
I'm also thinking that we have a max size for direct transfers and anything
larger goes through a temp file reguardless.
That's probably best and allows udp without streaming issues.

> > on my internet connection I'd quite like to switch them off entirely tbh
> 
> That could be an easy toggle command.

That'd be quite nice in any case. Do you think you can do that in parallel?

> > because, depending on the server,downloads happen either so fast I can't count the dots or so
> > slowly that having a random dot appearing is not very useful when I don't know
> > the actual file size.
> 
> I guess I don't understand the second part.
> Large downloads would happen in background, and you don't get the dots then anyways.

It depends really. It'd be nice to have some indication what's happening which
gave me information I could use, I can't really count the dots and yet
sometimes I need to load large documents (think pdf manuals,
yes some of them are huge) to actually view them.

> > Try counting how far through a 200 meg download you are in dot form,
> 
> It's easy, I just hit control w.
> Again, that's my adapter, and maybe why I like the feature so much.

Yes, that's adapter specific as far as I know,
and we really need edbrowse to be useful with or without it.

> > content-length header) and the actual amount (to the byte preferably)
> > that's actually been downloaded so I can see how long I have to wait.
> 
> sure, like a progress bar or percent indicator.
> Actually I like that less but could be done and selected by the user,
> dots or percent or quiet.

Yeah, sounds good. I'd probably print something like <amount downloaded>/<total
size> as a progress indicator rather than converting to percent but again
that's personal preference.

Cheers,
Adam.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]