New approach to grouping/backend/marking - good or bad idea?

New approach to grouping/backend/marking - good or bad idea?

[Rob Browning]

These may very well just be "bad ideas(TM)", but, after running them
by another Gnus notable and not getting "laughed out of the room", I'd
like to run them by whoever else will listen to see what they think.
Aside from possibly being just wrongheaded, these ideas might also be
completely unimplementable.  Since I don't have enough experience with
the Gnus codebase to evaluate if that's true easily, I figured I'd ask
those who do.

There are two bits here.  The bit that I feel pretty sure is a good
idea is that of adding "message keywords" to Gnus.  Imagine that you
can (either manually, or via automatic means like the split process),
add arbitrary keywords to articles.  These keywords could then be used
for whatever you like, but I could imeediately see a use for them in
searches, narrows, etc.  This would easily eliminate all the arguments
that keep me from being happy just using total-expire.

For example, I would like to use a "todo" keyword, and attach that to
articles that I need to do something about.  Then I could enter a
group and do a "/ k todo" or whatever, and see all those articles.

I could even see extending the system to support both user and system
level keywords (i.e. separate keyword namespaces).  The system level
keywords could be used to implement a more flexible "marking" system.
Things like read, expired, killed, could just become keywords.  This
would mean that you could easily support something like allowing an
article to have two marks, and, with a little more work, the user
could decide which of their kewords receive the "dormant behavior",
the "ticked behavior", etc.

I've also been toying with a the idea of a much more radical variation
on the same theme.  What if the mailer stored every message *only
once* in a guaranteed uniquely named file (across the entire backend),
and then used keywords (instead of physical placement in the
filesystem) to indicate groups?

In such a setup, a "group" could just become the set of articles which
matches a given keyword search criteria, crossposting an article would
just means adding more keywords to it, hardlinks wouldn't be needed
for efficiency, deciding when to delete an article would be (as it
should be) a completely orthogonal issue from deciding which groups it
is in, and, finally, since you have a guaranteed unique ID for every
article, you could easily use those article ID's like URLs to refer
back to important mail from other files.  You could safely say things
in your todo list like

  * Make sure to implement the frobbing mentioned in in <gnus://4126>
  * Don't forget the devices for the ritual <gnus://2311>

and then access these articles instantly via a nifty
gnus-jump-to-article function that worked from any emacs buffer.

There are also a couple of system level advantages to using this
approach.  One minor one is that you could use subdirectories
automagically for efficiency.  For example, you could store article
<gnus://23> as:

  Mail/storage/000/000/000/023

This way no directory would ever have more than 999 files.  As
compared to the current system where you can end up with thousands of
files in a single group, this would probably be a performance win,
though it's arguably one that would really be better solved at the OS
level.

Is this interesting?  Would it be too much work for too little gain?

Thanks

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: New approach to grouping/backend/marking - good or bad idea?

[Colin Marquardt]

* Rob Browning <rlb@cs.utexas.edu> writes:

> There are two bits here.  The bit that I feel pretty sure is a good
> idea is that of adding "message keywords" to Gnus.  Imagine that you
> can (either manually, or via automatic means like the split process),
> add arbitrary keywords to articles.  These keywords could then be used
> for whatever you like, but I could imeediately see a use for them in
> searches, narrows, etc.  This would easily eliminate all the arguments
> that keep me from being happy just using total-expire.

> For example, I would like to use a "todo" keyword, and attach that to
> articles that I need to do something about.  Then I could enter a
> group and do a "/ k todo" or whatever, and see all those articles.

Karl Kleinpaste has written
| ;;; messkeyw.el --- automatic keyword support during composition.
which is part of GNU Emacs (so it says).

| ;; This provides a hookable mechanism by which to have Keywords
| ;; headers automatically generated based on word frequency of the
| ;; body.  The goal is to make it possible to score on Keywords
| ;; provided, of course, that Keywords gets to the overview files.

I seem to remember that there was a HOWTO somewhere about adding the
keywords afterwards to exisiting articles.

This may be a start.

Colin

Re: New approach to grouping/backend/marking - good or bad idea?

[Karl Kleinpaste]

As Colin mentioned, I produced a module messkeyw.el for the purpose of
prefrobnicating keywords on articles, both news (when posting) and
mail (when sending as well as when receiving), simply by counting the
most common words found.  You can find it at
http://www.cs.cmu.edu/~karl/gnus/messkeyw.el.  Putting it to use well
requires that you do things with {gnus,nnmail}-extra-headers and then
rebuild your .overviews, if you use nnml.  Putting it to use *really*
well requires that you are a client of an INN installation which
supports Keywords in overviews, and (best of all) that the INN
incantation there has my related patch to auto-generate keywords on
all incoming article bodies (within size limitation, to avoid
pointlessly figuring keywords on MIMEd binaries and so forth).

I produced this module in order to manufacture an ability to score on
the Keywords header, which (under this type of implementation) happens
to be a particularly good indicator of conversation relevance, much
more so than scoring on things like Subject, because topic drift away
from advertised Subject is so routine.

FWIW, message-auto-keyword-insert in messkeyw.el generated the
following header from this piece of mail:
Keywords: keywords,system,gnus,article,articles,use,level,easily,used,todo,keyword,group,files
And since keyword issues interest me, I happen to score up on
"keywords" in Keywords, which meant that this piece of mail percolated 
nicely to the top of my ding *Summary*.

A lot can be done with Keywords.  But adding a neater interface for
tweaking header content would certainly be interesting and useful; as
it is, you can always `C-u g e' a mail message and add keywords of any
sort that makes you feel good, including "todo" or whatnot.  As to a
quick jump-to operation based on such stuff, you're probably stepping
farther into unknown water than you may realize, but on the other
hand, it would be a cool additional interface.

As to your suggestion for generalized storage...no, please don't.  One 
of the benefits of per-group directory storage is that I can simply
step into a directory and grep for something interesting.  Yes, I have 
nnir.el (and use it heavily -- it has become part of my work here) but 
there is still benefit ultimately in being able to ask truly simple
UNIX questions of my filesystem.

--karl

PS- The comment in messkeyw.el about "part of GNU Emacs" is because it
was intended that messkeyw.el be inhaled as part of Gnus, but Lars has
never quite taken it to that point.  Ohwell.

Re: New approach to grouping/backend/marking - good or bad idea?

[Rob Browning]

Karl Kleinpaste <karl@justresearch.com> writes:

> A lot can be done with Keywords.  But adding a neater interface for
> tweaking header content would certainly be interesting and useful; as
> it is, you can always `C-u g e' a mail message and add keywords of any
> sort that makes you feel good, including "todo" or whatnot.

All this sounds interesting.  I'll have to look in to it more
carefully...

> As to your suggestion for generalized storage...no, please don't.
> One of the benefits of per-group directory storage is that I can
> simply step into a directory and grep for something interesting.
> Yes, I have nnir.el (and use it heavily -- it has become part of my
> work here) but there is still benefit ultimately in being able to
> ask truly simple UNIX questions of my filesystem.

Well, I envisioned that there would be a shell level tool that would
allow you to generate file lists i.e.

  gnus-find -k debian-devel -a -k -debian-user -not -k debian-policy

or whatever.

So you should be able to do nearly the same things you can do with
separate directories now, including exporting all your stuff as
separate directories if you want.  However, I will certainly concede
that this isn't as straightforward as just having a set of
directories.  I'm sure it makes some things easier and some things
harder.  I just started to think that I might want my mail to be more
like a database than just a collection of files (though I'll always
want to be able to get at it as files).

In any case, I'm beginning to think that overall, for now, I may just
need to switch over to total-expire (if I plan to keep using
crossposting i.e. "&" splits), and just get used to it.  It's too
weird to have an article that's marked as expire in one group get the
read mark in all the others.

And although I'd rather use the expiry process, I suppose I can just
use 'B del' when I want to remove a crossposted article from the
current group, but not affect it elsewhere, though I'd still like to
have control over what happens to marks on crossposts.  And finally,
unless I physically renumber my articles, I think I can always use the
filesystem location as a "permanent pointer".

I'll have to think about that for a bit...

Thanks

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: New approach to grouping/backend/marking - good or bad idea?

[Kai Großjohann]

Karl Kleinpaste <karl@justresearch.com> writes:

> As to your suggestion for generalized storage...no, please don't.  One 
> of the benefits of per-group directory storage is that I can simply
> step into a directory and grep for something interesting.  Yes, I have 
> nnir.el (and use it heavily -- it has become part of my work here) but 
> there is still benefit ultimately in being able to ask truly simple
> UNIX questions of my filesystem.

I think it might be of benefit if Rob just tried to implement his idea
as a backend.  This would require him to provide some semblance of
`groups' to the rest of Gnus; but I don't think that this is much of a
problem -- if you take the groups away from Gnus, what have you left?
Not much, anyway.

I think it is extremely unlikely that Gnus will change such that the
group concept disappears.  (Groups are needed for news, for example.)

And the backend interface might be flexible enough.  Maybe some
additional part of Gnus needs to be updated, but I think you can get
pretty far with just a backend.

And once the idea begins to take shape, we can look at it and think
about it.

But OTOH, maybe it is sufficient to couple existing Gnus features such
as your messkw and gnus-extra-headers and limiting.  Hm.  I don't know.

kai
-- 
This gubblick contains many nonsklarkish English flutzpahs,
but the overall pluggandisp can be glorked from context. -- David Moser

Re: New approach to grouping/backend/marking - good or bad idea?

[Russ Allbery]

Karl Kleinpaste <karl@justresearch.com> writes:

> Putting it to use *really* well requires that you are a client of an INN
> installation which supports Keywords in overviews, and (best of all)
> that the INN incantation there has my related patch to auto-generate
> keywords on all incoming article bodies (within size limitation, to
> avoid pointlessly figuring keywords on MIMEd binaries and so forth).

Which reminds me... getting that to build properly with current versions
of INN involves some very bizarre incantations, but I'll try to clean that
up and make it more straightforward before INN 2.3 is released.

-- 
Russ Allbery (rra@stanford.edu)         <URL:http://www.eyrie.org/~eagle/>

Re: New approach to grouping/backend/marking - good or bad idea?

[Lars Magne Ingebrigtsen]

Rob Browning <rlb@cs.utexas.edu> writes:

> For example, I would like to use a "todo" keyword, and attach that to
> articles that I need to do something about.  Then I could enter a
> group and do a "/ k todo" or whatever, and see all those articles.

This is possible now with the Keywords header, extra headers and `/ x'.
It's a bit of cumbersome to set up, though.

> I've also been toying with a the idea of a much more radical variation
> on the same theme.  What if the mailer stored every message *only
> once* in a guaranteed uniquely named file (across the entire backend),
> and then used keywords (instead of physical placement in the
> filesystem) to indicate groups?

This sounds like a job for nnsql.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen