[Edbrowse-dev] mailx

[Edbrowse-dev] mailx

[Karl Dahlke]

> I think that the heirloom-mailx version can do all these things since it
> supports every mail protocol I can think of.

It's not the protocols, it's the user interface.

> we need ...  imap (with imaps and starttls support),

Handled by curl, to some degree.

> pop3 (with ssl and tls support),

Already done, mostly by curl.

> mbox and maildir support (for local mail),

I wouldn't use this; keep mail on the server or put mails
on your computer where they really belong.

> Correct threading header support (currently it doesn't work correctly,

It does, but you have to save email unformatted and browse and re.
I need to make this mechanism more convenient from a user perspective,
but it is implemented inside.
I had to do this a few years ago because people on one of the linux
lists yelled at me, each of my emails started a new thread
and confused them and I really meant it to be a reply on a current thread
so anyways yes I did some of this work.
It wasn't hard.
I think it's documented somewhere.

> Other stuff ...

Yes, not sure how vital the other stuff is.

> but I wonder if it's time for an edmail program?

We can certainly see the advantages of the separate eb js process,
now that it's done,
and ebmail ebspread etc could make a lot of sense, as long as it's all an integrated system.
Bill Gates found this out a long time ago.
Some web forms call up email client so you can send email,
or even submit form by email,
and from within email you can click on a link and go to your browser,
and (not speaking of windows but of edbrowse)
I can be in the editor and suddenly decide I want to send somebody email
and just put subject at the top and type sm and out it flies.
I really like that.
I really like the integratedness of it all,
but that certainly doesn't preclude modularizing the software more than it is today.

Karl Dahlke

Re: [Edbrowse-dev] mailx

[Adam Thompson]

[-- Attachment #1: Type: text/plain, Size: 4224 bytes --]

On Sun, Apr 12, 2015 at 12:00:47PM -0400, Karl Dahlke wrote:
> > we need ...  imap (with imaps and starttls support),
> 
> Handled by curl, to some degree.

Thought so, but it's the "to some degree" part which concerns me.
> > pop3 (with ssl and tls support),
> 
> Already done, mostly by curl.

I was fairly sure this was the case.

> > mbox and maildir support (for local mail),
> 
> I wouldn't use this; keep mail on the server or put mails
> on your computer where they really belong.

You wouldn't but, if we want to make this more than a project used by only a
hand full of people, we need this support.
I personally need to access my local mbox because I have things like cron jobs
which occasionally fail and their emails land in my local mbox.
Also, I used to run an email server (and know people who do the same)
which could receive email from the internet (and had the relevant dns records
set up to make this happen) but for which the method of email access was
maildir via a local email client. At the end of the day,
edbrowse is the *only* email client I know which doesn't do this.
Well the only Linux one at least.

> > Correct threading header support (currently it doesn't work correctly,
> 
> It does, but you have to save email unformatted and browse and re.
> I need to make this mechanism more convenient from a user perspective,
> but it is implemented inside.
> I had to do this a few years ago because people on one of the linux
> lists yelled at me, each of my emails started a new thread
> and confused them and I really meant it to be a reply on a current thread
> so anyways yes I did some of this work.
> It wasn't hard.
> I think it's documented somewhere.

Could you possibly demonstrate this by *not* starting a new thread when you
reply to this email (and all emails actually);
Put me down as one of those users who'll start yelling at you for this.
It's quite annoying when using a client like mutt, which does threading, when
your replies appear out of order with the rest of the thread.
Actually, add threading to the requirements for an ebmail program.

> > Other stuff ...
> 
> Yes, not sure how vital the other stuff is.

Probably not very, but it's worth designing things in such a way that additions won't be too complex.

> > but I wonder if it's time for an edmail program?
> 
> We can certainly see the advantages of the separate eb js process,
> now that it's done,
> and ebmail ebspread etc could make a lot of sense, as long as it's all an integrated system.
> Bill Gates found this out a long time ago.
> Some web forms call up email client so you can send email,
> or even submit form by email,
> and from within email you can click on a link and go to your browser,
> and (not speaking of windows but of edbrowse)
> I can be in the editor and suddenly decide I want to send somebody email
> and just put subject at the top and type sm and out it flies.
> I really like that.
> I really like the integratedness of it all,
> but that certainly doesn't preclude modularizing the software more than it is today.

I'd probably go for a system whereby users could define commands,
thus you could define the sm command to use ebmail,
whereas I'd probably use mutt (until ebmail becomes a usable email client for me).
I'd definitely define something to launch (what'd the browser be called in this
naming system?) on a URL though. Basically what I'm proposing is a system which
can be as integrated as *you* want, but flexable enough for users like *me* to
make our own choices as to the programs we use to carry out certain operations.
I'm trying to design away from the single does everything for everyone model of
Windows (and to some extent some of Gnome etc on Linux),
and more towards the provides tools to do everything model of *nix.
Actually, I think even Windows provides ways for you to override the defaults for things.
Personally, if ebmail can be as fully featured as what I've currently got then
I'd really like to use it, but I'd like options and don't particularly want to
get into designing separate programs which are, under the surface,
very tightly coupled.

Cheers,
Adam.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Edbrowse-dev] mailx, plugin commands, mime, new sourcefile

[Karl Dahlke]

> Are we using a head request for http to determine this?

I never issue a head request, primarily because I don't need to 95%
of the time, and I never know when that other 5% is coming.
You pointed out, rightly so, that foobar.php can crank out
anything, usually it's just another web page,
wherein head would be a complete waste of time,
and maybe quite a bit of time if that php program did a lot of computatio
to get you to that next web page.
Maybe it even ordered something for you through amazon,
and issuing the command twice would order the item twice.
I don't know but messing with head commands scares me.

> I ask as in some situations I can think of,
> re-invoking the player with the URL once downloading has began may break things

Yeah I get that. That could happen.
Not sure what to say about that.
The edbrowse abort happens during the headers, not after,
and that's a worst case,
assuming it isn't redirected earlier by protocol,
but if/when this happens
most likely it would just look like a reconnect and not cause any trouble.
It's even a connect from another program, another local port,
so probably wouldn't be a problem.

> I'd also be in favour of some sort of protocol set up
> which would hand off URLs to external programs,
> for example rtsp may be handed to mplayer for playing.

This is already done and has worked for quite some time.
Look at this mime descriptor from my config file.
It's also in the sample config file in the doc directory.

mime {
#  the < forces it to be a stream, hence the url is passed to the program directly
type = <audio/x-pn-realaudio
desc = streaming audio
suffix = rm,ra,ram,pls
content = audio/x-scpls
protocol = rtsp,pnm,sdp
program = /usr/bin/mplayer -quiet
}

The content = attribute is supported with the latest version,
that wasn't there before,
but it's there now and mostly works, and the other stuff worked before
and would pass the url along based on protocol or suffix.
This reminds me I need to reinstall mplayer, it's not on my new system.
Guess I don't listen to real audio very often.
I never got nasa real audio to run through mplayer,
and that bums me out cause it's not on cable any more and I
don't have any way to listen to space launches or coverage etc.
But I digress.

> I'd also like a pga option for plugins ask

Yes that's a good thought.
We should think about this one for a bit.

I added mime.c, a new sourcefile, because well it just makes sense.
I couldn't make any progress with little chunks of mime and plugin code
scattered all through the files.
So I gathered them together in mime.c, almost 300 lines of code,
and it will grow quite a bit I'm sure as we develop this important capability.
git pull and you'll see the new sourcefile.

The behavior of edbrowse is still mostly as it was before,
I've just made foundational changes,
but now I can make some real changes / improvements.

> Actually, I think even Windows provides ways for you to override the defaults for things.

Sure they do, now, after the law suit,
so they have to allow users to use maybe explorer + eudora, or perhaps firefox + outlook.
Naturally they wanted to keep you locked into their software only and forever,
but that wouldn't fly.
But I digress.

> Could you demonstrate this by *not* starting a new thread when you
> reply to this email (and all emails actually);

Ok this is a test case, it should thread up with your mailx email.
I did change the subject but I don't think that matters.
If this worked I'll make it part of my regular procedure for communication
on this list.

Karl Dahlke

Re: [Edbrowse-dev] mailx, plugin commands, mime, new sourcefile

[Adam Thompson]

[-- Attachment #1: Type: text/plain, Size: 5031 bytes --]

On Sun, Apr 12, 2015 at 04:46:08PM -0400, Karl Dahlke wrote:
> > Are we using a head request for http to determine this?
> 
> I never issue a head request, primarily because I don't need to 95%
> of the time, and I never know when that other 5% is coming.
> You pointed out, rightly so, that foobar.php can crank out
> anything, usually it's just another web page,
> wherein head would be a complete waste of time,
> and maybe quite a bit of time if that php program did a lot of computatio
> to get you to that next web page.
> Maybe it even ordered something for you through amazon,
> and issuing the command twice would order the item twice.
> I don't know but messing with head commands scares me.

Fair enough, it's tricky and on further reflection I can see the validity of
your arguments.
> > I ask as in some situations I can think of,
> > re-invoking the player with the URL once downloading has began may break things
> 
> Yeah I get that. That could happen.
> Not sure what to say about that.
> The edbrowse abort happens during the headers, not after,
> and that's a worst case,
> assuming it isn't redirected earlier by protocol,
> but if/when this happens
> most likely it would just look like a reconnect and not cause any trouble.
> It's even a connect from another program, another local port,
> so probably wouldn't be a problem.

May be, it depends on how the content distribution network works. I think we're usually ok with our current mechanism so that's fine with me.

> > I'd also be in favour of some sort of protocol set up
> > which would hand off URLs to external programs,
> > for example rtsp may be handed to mplayer for playing.
> 
> This is already done and has worked for quite some time.
> Look at this mime descriptor from my config file.
> It's also in the sample config file in the doc directory.
> 
> mime {
> #  the < forces it to be a stream, hence the url is passed to the program directly
> type = <audio/x-pn-realaudio
> desc = streaming audio
> suffix = rm,ra,ram,pls
> content = audio/x-scpls
> protocol = rtsp,pnm,sdp
> program = /usr/bin/mplayer -quiet
> }
> 
> The content = attribute is supported with the latest version,
> that wasn't there before,
> but it's there now and mostly works, and the other stuff worked before
> and would pass the url along based on protocol or suffix.
> This reminds me I need to reinstall mplayer, it's not on my new system.
> Guess I don't listen to real audio very often.
> I never got nasa real audio to run through mplayer,
> and that bums me out cause it's not on cable any more and I
> don't have any way to listen to space launches or coverage etc.
> But I digress.

Thanks for pointing out the protocol mechanism;
I see that most of the attributes aren't necessary,
but I'd like it if the type attribute was optional in the case that the
protocol attribute is provided.
I'd also probably rename this as plugin since it's now more than just mime type handlers.
For example:
plugin {
protocol=rtmp
program=flvstreamer
desc=rtmp streaming content
}
(I'm not sure about the exact flvstreamer options to get this to do anything sensible).
In this case type doesn't really apply, but name would and so does description.

> > I'd also like a pga option for plugins ask
> 
> Yes that's a good thought.
> We should think about this one for a bit.

Thanks, you're right this needs thought.

> I added mime.c, a new sourcefile, because well it just makes sense.
> I couldn't make any progress with little chunks of mime and plugin code
> scattered all through the files.
> So I gathered them together in mime.c, almost 300 lines of code,
> and it will grow quite a bit I'm sure as we develop this important capability.
> git pull and you'll see the new sourcefile.

Should this be plugin.c rather than mime.c?

> The behavior of edbrowse is still mostly as it was before,
> I've just made foundational changes,
> but now I can make some real changes / improvements.

Sounds good.

> > Actually, I think even Windows provides ways for you to override the defaults for things.
> 
> Sure they do, now, after the law suit,
> so they have to allow users to use maybe explorer + eudora, or perhaps firefox + outlook.
> Naturally they wanted to keep you locked into their software only and forever,
> but that wouldn't fly.
> But I digress.

Can we avoid the same locking in of users? I'd prefer it if we could.

> > Could you demonstrate this by *not* starting a new thread when you
> > reply to this email (and all emails actually);
> 
> Ok this is a test case, it should thread up with your mailx email.
> I did change the subject but I don't think that matters.
> If this worked I'll make it part of my regular procedure for communication
> on this list.

Yes, this works fine in my email client,
though the change of subject confused the googlemail web interface (which
apparently doesn't quite do threading properly).

Cheers,
Adam.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Edbrowse-dev] mailx, plugin commands, mime, new sourcefile

[Karl Dahlke]

> but I'd like it if the type attribute was optional

I was looking at this stuff and asked myself, "what the hell is type for?"
It is used nowhere, except one place, where the string
is copied into a mime object in the js world.
There is such an object for each mime entity in the config file.
That object wants a type attribute, I don't know what for.
We could probably do without it just fine,
or maybe just copy the description in there.

> I'd also probably rename this as plugin since it's now more than just mime type handlers.

I think that's a great idea, will probably do that
unless someone objects.
Not sure if I will retroactively change all the variables in the source code,
but at least the config file syntax.

> Should this be plugin.c rather than mime.c?

Yes I think so, I'll do a git rename soon.
That's easy. Thanks.


Karl Dahlke

Re: [Edbrowse-dev] mailx, plugin commands, mime, new sourcefile

[Adam Thompson]

[-- Attachment #1: Type: text/plain, Size: 1433 bytes --]

On Mon, Apr 13, 2015 at 04:33:57PM -0400, Karl Dahlke wrote:
> > but I'd like it if the type attribute was optional
> 
> I was looking at this stuff and asked myself, "what the hell is type for?"
> It is used nowhere, except one place, where the string
> is copied into a mime object in the js world.
> There is such an object for each mime entity in the config file.
> That object wants a type attribute, I don't know what for.
> We could probably do without it just fine,
> or maybe just copy the description in there.

I'd probably put the content type in there actually since that seems the most
logical thing to go into the type attribute of a mime object to me.
If this is unspecified then just don't create a mime object for that plugin.

> > I'd also probably rename this as plugin since it's now more than just mime type handlers.
> 
> I think that's a great idea, will probably do that
> unless someone objects.
> Not sure if I will retroactively change all the variables in the source code,
> but at least the config file syntax.

Thanks, I think the variables can be altered either now or later depending on
how these plugins are used in a given context, e.g.
in some situations the mime naming may make sense.

> > Should this be plugin.c rather than mime.c?
> 
> Yes I think so, I'll do a git rename soon.
> That's easy. Thanks.

As well as altering the Makefiles etc.

Cheers,
Adam.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Edbrowse-dev] mailx

[Karl Dahlke]

Usually I refrain from commenting about other systems, other worlds,
because I've never used them.
Never touched windows, or gnome, or kde, or firefox,
or speakup, or just about anything else; I live in my own little world.
But aha, I did use mailx.
It must have been nearly 30 years ago at bell labs.
I liked it, and could still use it, but it entails a 2 pass process.
I pulled mail down, and perhaps stored it,
but then later I had to do more things with my emails.
I wrote edbrowse to be an integrated system.
Each email I can delete, or store, and put this attachment here,
and throw that attachment away because it's just an image, and so on.
One pass and I manage all the new emails and I'm done.
That's why I typically have 6 emails pending, while my wife has 2000.
As a side benefit, the rendered email looks just like it would
if I edited the email later, in edbrowse, and browsed.
This is more and more important as almost all emails
are in html.
So the first time I see it, when I'm wondering what to do with it,
or even if I should keep it at all,
it has the familiar braces and formatting of an edbrowse page,
just like it will the next time I see it.
All these points are small, and yet they're not small in that I
fetch mail dozens of times a day and would like it to be ultra
efficient and convenient.
Even though I liked mailx, I will never go back to it.
With that in mind, I think a simple version of imap could be done in a few hundred lines of code,
with no structural changes and no negative immpact to edbrowse,
if you never used imap it would never bother you, etc,
and I could use it in a simple manner if I wished.
The day may come when large mail servers only offer imap or exchange, no pop3 available.
gmail for example still offers pop3 but I had to go
into the settings section and set it up, because it is not
available by default.
So I may some day have to forget edbrowse as a mail client or support some level of imap,
and this may be what others are seeing,
and thus I would like a little bit of imap to work.
Chris you said it looked a little more involved than you had hoped;
if you don't want to mess with it I can look into it,
but I would need to know where you get all this cool information about 
curl development. It's probably a thick manual, hopefully with examples.
I'm thinking we should at least, as a first step,
call the inner mail management routine that we already wrote,
on the inbox folder, and get a feel for how that works.
We could then build from there if we like it,
or discard the effort if we don't.

Karl Dahlke

Re: [Edbrowse-dev] mailx

[Chris Brannon]

Karl Dahlke <eklhad@comcast.net> writes:

> but I would need to know where you get all this cool information about 
> curl development. It's probably a thick manual, hopefully with examples.

Actually no.  Much of the library is documented with manpages, and
they are generally quite good.
But the POP and IMAP stuff isn't so well-documented.
There's some example source code, and that's about it.
The problem with IMAP support in curl is that we have to manually parse
a lot of responses that we get, for instance the response to a "list folders"
or "list messages" command.
We don't get the data in a form we can use directly.
But I wonder if we couldn't borrow code from mailx.  The version that
supports IMAP is called "heirloom-mailx", formerly known as "nail", and
it is licensed under a permissive BSD license.  So it'd be fine to
incorporate code from that project.

-- Chris

Re: [Edbrowse-dev] mailx

[Adam Thompson]

[-- Attachment #1: Type: text/plain, Size: 4034 bytes --]

On Sat, Apr 11, 2015 at 01:19:50PM -0400, Karl Dahlke wrote:
> Usually I refrain from commenting about other systems, other worlds,
> because I've never used them.
> Never touched windows, or gnome, or kde, or firefox,
> or speakup, or just about anything else; I live in my own little world.
> But aha, I did use mailx.
> It must have been nearly 30 years ago at bell labs.
> I liked it, and could still use it, but it entails a 2 pass process.
> I pulled mail down, and perhaps stored it,
> but then later I had to do more things with my emails.
> I wrote edbrowse to be an integrated system.
> Each email I can delete, or store, and put this attachment here,
> and throw that attachment away because it's just an image, and so on.
> One pass and I manage all the new emails and I'm done.
> That's why I typically have 6 emails pending, while my wife has 2000.

I think that the heirloom-mailx version can do all these things since it
supports every mail protocol I can think of.
I'm going to investigate this since I rather like the interface and often end
up wanting to edit email by sshing into a remote machine and ed-like
interfaces just work better over slow connections in my opinion.

> As a side benefit, the rendered email looks just like it would
> if I edited the email later, in edbrowse, and browsed.
> This is more and more important as almost all emails
> are in html.
> So the first time I see it, when I'm wondering what to do with it,
> or even if I should keep it at all,
> it has the familiar braces and formatting of an edbrowse page,
> just like it will the next time I see it.

Yeah I can see that being quite nice;
though you could just use the mailcap mechanism to do this of course.

> All these points are small, and yet they're not small in that I
> fetch mail dozens of times a day and would like it to be ultra
> efficient and convenient.
> Even though I liked mailx, I will never go back to it.
> With that in mind, I think a simple version of imap could be done in a few hundred lines of code,
> with no structural changes and no negative immpact to edbrowse,
> if you never used imap it would never bother you, etc,
> and I could use it in a simple manner if I wished.
> The day may come when large mail servers only offer imap or exchange, no pop3 available.
> gmail for example still offers pop3 but I had to go
> into the settings section and set it up, because it is not
> available by default.
> So I may some day have to forget edbrowse as a mail client or support some level of imap,
> and this may be what others are seeing,
> and thus I would like a little bit of imap to work.

I guess I'm just a litle worried about bloat as the browsing side of edbrowse
gets progressively larger in order to keep up with the modern internet.
In addition, there are many features we simply don't have in edbrowse,
I'm not sure exactly what the mail support is,
but for vague parity with modern clients we really need:
imap (with imaps and starttls support),
pop3 (with ssl and tls support),
mbox and maildir support (for local mail),
support for pgp (and for some people s/mime support as well),
Correct threading header support (currently it doesn't work correctly, at least from what I can see in the mail headers in emails I receive from you),
Mailcap support of some kind (potentially our plugin system may be able to do
this but it should probably be integrated),
Probably a bunch of other stuff I've never used.

I'm not saying we can't do this in edbrowse, but I wonder if it's time for an edmail program?

If we modularise the buffer and command parsing interface we could have an ed
suite, with edmail, edbrowse, edtext and (if you want) eddb and edspread.
This would make things quite a lot nicer to work on I suspect,
whilst also making adding new programs (e.g. edspread) that much easier.
There are also other projects where I'd quite like the buffer and command
parsing (and readline) stuff from edbrowse.
A hex editor would be one which would be quite nice to investigate.

Cheers,
Adam.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]