Separating marks from other group configuration

Separating marks from other group configuration

[Lars Magne Ingebrigtsen]

While making dinner here I was thinking about what Steinar said about
separating out the non-marks stuff (group parameters and...  stuff?)
from .newsrc.eld.  Which led me to thinking about where to store it.
Which led me to thinking about IMAP.

Wouldn't it be nice -- if you had an IMAP server in your setup, then
everything would be stored there?  (I mean, in addition to the
.newsrc.eld file, which would be used if you couldn't access the IMAP
server.)

The marks are already on the server.  So if Gnus just uploaded the rest
of the data (say, as an article to a special "gnus-config" mailbox
there), then your Gnus setup would be totally mobile...  And in the
cloud!  It would be cloudy!

Now I have to finish dinner before things burn.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Separating marks from other group configuration

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Wouldn't it be nice -- if you had an IMAP server in your setup, then
> everything would be stored there?  (I mean, in addition to the
> .newsrc.eld file, which would be used if you couldn't access the IMAP
> server.)

Yep.

Ted has already had some thoughts in this line, about extending his
gnus-sync.el to cover stuff other than tick marks.

Somewhere in this group.  Probably easily searchable with `G G' :-)

Re: Separating marks from other group configuration

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Ted has already had some thoughts in this line, about extending his
> gnus-sync.el to cover stuff other than tick marks.

That's what so good about having such a crappy memory like I have.  I
think of something, and then imagine that they're my ideas instead of
something I read a few days ago.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

gnus-sync ideas (was: Separating marks from other group configuration)

[Ted Zlatanov]

On Mon, 13 Dec 2010 19:49:00 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:
>> Wouldn't it be nice -- if you had an IMAP server in your setup, then
>> everything would be stored there?  (I mean, in addition to the
>> .newsrc.eld file, which would be used if you couldn't access the IMAP
>> server.)

SB> Ted has already had some thoughts in this line, about extending his
SB> gnus-sync.el to cover stuff other than tick marks.

OK, let's throw some gnus-sync specific ideas around.  Assume the
current implementation is just a prototype and can be entirely thrown
away.  Also assume I will work on this soon-ish but am happy to let
others contribute up to 100%.  Finally, let's say one of the goals is to
eliminate newsrc.eld and other Gnus config files entirely or at least
make them a gnus-sync storage backend.

First of all, how should it operate in Gnus?  IMO it should be immediate
and not triggered as it is now; a sync should happen every time you
enter or exit a group (and can also be triggered explicitly).  Here we
can use the hooks I added recently: `gnus-after-set-mark-hook' and
`gnus-before-update-mark-hook'.

Second, how should the data be stored?  IMO it should be stored in a
free-form key-value format but the storage backend should be able to
understand the rtree format; the ELisp list, vector, and plist formats;
and maybe JSON and merge such data (with parameters that control how the
merge will happen).  I think a custom network server would actually be
the best storage backend solution here; imap-hash.el could also work
with some love; the file storage backend is also sensible to duplicate
the current newsrc.eld functionality; and, of course, people will come
up with a Git-based and RDBMS-based storage backend Because They Can.

Third, what data are we storing?  I think marks should only be synced
for news-p backends.  But we should generally be able to save, in order
of importance IMHO:

- marks for news-p servers (medium difficulty)

- splitting rules (trivial but needs a view/edit interface)

- BBDB (hard but very important, and useful beyond Gnus)

- topic hierarchy (medium)

- subscriptions (but the user can pick subsets as the "work" and "home"
  setup for instance) (hard)

- the Gnus registry (medium)

So because this is such diverse data I think the storage backend should
just be a key-value store.  The rtree/ELisp data merge support is really
the only thing special about it.  And maybe it's not a Gnus-specific
thing but could be useful for Emacs in general, in which case it should
be called data-sync.el and the Gnus glue is gnus-sync.el.  I'm sure the
org-mode users, for instance, would like this kind of functionality too,
and maybe they already have it and we can just steal it :)

Fourth, for general accessibility, data-sync.el will have a simple
view/edit interface to load a specific key in a buffer, edit it in Emacs
Lisp, rtree, or JSON mode, and commit it back if it validates (no merge,
just overwrite).  This functionality could also connect with revisions
if the storage backend supports them, so you could look at older
versions of your data.

Fifth, the network server storage backend should be easy to set up and
run.  Maybe a HTTP-backed implementation would make the most sense
because those are so easy to set up and are not firewalled.

Finally, what other parts of Emacs besides BBDB and org-mode could
benefit from a general data-sync facility?

WDYT?

Ted

Re: gnus-sync ideas

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> And maybe it's not a Gnus-specific thing but could be useful for Emacs
> in general, in which case it should be called data-sync.el and the
> Gnus glue is gnus-sync.el.

That's an intriguing idea.  I mean, Emacs-in-the-cloud, sort of.  So
when I start up my Emacs on my MeeGo phone (or something; my phone ain't
smart enough yet), then it'd pull down all the, er, relevant bits to
allow me to read news/mail/etc with the least amount of stored data
locally.

However, this has to work unplugged too, and it should sync (with the
least amount of data transferred) when it manages to connect to the
network.  So Emacs should have a way of asking "what's happened since
<TIMESTAMP>" and get all the changes...

That'd be cool.

The challenge (in addition to this being generally a challenging problem
area :-) would be to make this fast enough that it'd be appealing...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-sync ideas

[Ted Zlatanov]

On Wed, 15 Dec 2010 20:18:50 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> And maybe it's not a Gnus-specific thing but could be useful for Emacs
>> in general, in which case it should be called data-sync.el and the
>> Gnus glue is gnus-sync.el.

LMI> That's an intriguing idea.  I mean, Emacs-in-the-cloud, sort of.

Yeah, the key is that the server can *merge* ELisp data structures and
possibly take partial updates.  Without that functionality, it's no
better than a Tramp remote file.

LMI> So when I start up my Emacs on my MeeGo phone (or something; my
LMI> phone ain't smart enough yet), then it'd pull down all the, er,
LMI> relevant bits to allow me to read news/mail/etc with the least
LMI> amount of stored data locally.

Right.  So you'd rely on data-sync.el to do the data management while
gnus-sync.el would do the Gnus-specific control, verification,
and conversion.

LMI> However, this has to work unplugged too, and it should sync (with the
LMI> least amount of data transferred) when it manages to connect to the
LMI> network.  So Emacs should have a way of asking "what's happened since
LMI> <TIMESTAMP>" and get all the changes...

LMI> That'd be cool.

LMI> The challenge (in addition to this being generally a challenging problem
LMI> area :-) would be to make this fast enough that it'd be appealing...

I can do a lot of the server-side work required here.  Something with a
HTTP front-end that takes POST requests, linked to a Membase or SQLite
database, would be ridiculously fast (probably faster than the
newsrc/registry save mechanism, heh).  In fact I imagine you or me will
end up running some version of this as a service for Emacs users.

But I want to figure out the server/data-sync.el requirements before
writing even one line of the implementation because adding more features
later will be a pain.

First of all, the data types will be: ELisp rtree, list, plist, alist,
vector, hashtable; JSON array and map; string.

So, data-sync.el would let you:

1) choose a storage backend, which can be many things but will at least
have the choice to be a file (which, of course, can use Tramp).

2) query a key and find out the latest value, data type, CAS key, and
timestamp (you'd use the CAS key in the CAS ops below)

3) set a key with a data type (unconditionally or CAS)

4) delete a key (unconditionally or CAS)

5) merge data into a key with a data type and a policy (overwrite,
prepend, append, set-if-present, set-if-absent) (unconditionally or CAS)

6) do all of the above as a batch

7) synchronize a bunch of keys and data according to a given policy and
return which ones succeeded, which ones failed to merge, and which ones
failed due to a CAS key mismatch (so you need to retry them).  This
would be your basic API.

Does that sound reasonable?  What else would you need?

Ted

Re: gnus-sync ideas

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> So, data-sync.el would let you:
>
> 1) choose a storage backend, which can be many things but will at least
> have the choice to be a file (which, of course, can use Tramp).
>
> 2) query a key and find out the latest value, data type, CAS key, and
> timestamp (you'd use the CAS key in the CAS ops below)
>
> 3) set a key with a data type (unconditionally or CAS)
>
> 4) delete a key (unconditionally or CAS)
>
> 5) merge data into a key with a data type and a policy (overwrite,
> prepend, append, set-if-present, set-if-absent) (unconditionally or CAS)
>
> 6) do all of the above as a batch

If there's one thing that the nnimap experience shows, then it's that
round trips kill you.  So ideally the client would issue just one
command, like

GIVE-ME-DATA-CHANGED-SINCE 12167121621 <filter that says what data I want>

and then we'd just get a stream of data back.

So all CRUD operations to the store has to be timestamped and
categorised for this to be fast, I think. 

As for how to actually store things like, say, the group topics...  I
don't really have a clear idea how that would actually look...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-sync ideas

[Ted Zlatanov]

On Thu, 16 Dec 2010 17:08:07 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> So, data-sync.el would let you:
>> 
>> 1) choose a storage backend, which can be many things but will at least
>> have the choice to be a file (which, of course, can use Tramp).
>> 
>> 2) query a key and find out the latest value, data type, CAS key, and
>> timestamp (you'd use the CAS key in the CAS ops below)
>> 
>> 3) set a key with a data type (unconditionally or CAS)
>> 
>> 4) delete a key (unconditionally or CAS)
>> 
>> 5) merge data into a key with a data type and a policy (overwrite,
>> prepend, append, set-if-present, set-if-absent) (unconditionally or CAS)
>> 
>> 6) do all of the above as a batch

LMI> If there's one thing that the nnimap experience shows, then it's that
LMI> round trips kill you.  So ideally the client would issue just one
LMI> command, like

LMI> GIVE-ME-DATA-CHANGED-SINCE 12167121621 <filter that says what data I want>

LMI> and then we'd just get a stream of data back.

LMI> So all CRUD operations to the store has to be timestamped and
LMI> categorised for this to be fast, I think. 

All of the above can be batched, yes, and the query will have just one
timestamp instead of one per key.

LMI> As for how to actually store things like, say, the group topics...  I
LMI> don't really have a clear idea how that would actually look...

ELisp, of course.  It's just like reading them back from the newsrc.eld file.

Ted

Re: gnus-sync ideas

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 4426 bytes --]

On Wed, Dec 15 2010, Ted Zlatanov wrote:

> OK, let's throw some gnus-sync specific ideas around.  Assume the
> current implementation is just a prototype and can be entirely thrown
> away.  Also assume I will work on this soon-ish but am happy to let
> others contribute up to 100%.  Finally, let's say one of the goals is to
> eliminate newsrc.eld and other Gnus config files entirely or at least
> make them a gnus-sync storage backend.

So far, so good. But I know nothing about current implementation.

> First of all, how should it operate in Gnus?  IMO it should be immediate
> and not triggered as it is now; a sync should happen every time you
> enter or exit a group (and can also be triggered explicitly).  Here we
> can use the hooks I added recently: `gnus-after-set-mark-hook' and
> `gnus-before-update-mark-hook'.

Fair enough.

> Second, how should the data be stored?  IMO it should be stored in a
> free-form key-value format but the storage backend should be able to
> understand the rtree format; the ELisp list, vector, and plist formats;
> and maybe JSON and merge such data (with parameters that control how the
> merge will happen).  I think a custom network server would actually be
> the best storage backend solution here; imap-hash.el could also work
> with some love; the file storage backend is also sensible to duplicate
> the current newsrc.eld functionality; and, of course, people will come
> up with a Git-based and RDBMS-based storage backend Because They Can.

Thanks, I use git, so storing it inside git would be good for me, and
probably the simplest way. :)

> Third, what data are we storing?  I think marks should only be synced
> for news-p backends.  But we should generally be able to save, in order
> of importance IMHO:
>
> - marks for news-p servers (medium difficulty)

That's the only thing that really matters IHMO.

> - splitting rules (trivial but needs a view/edit interface)

I do not see why it's not a configuration (elisp) thing.

> - BBDB (hard but very important, and useful beyond Gnus)

I do not see the point.

> - topic hierarchy (medium)

I do not see why it's not a configuration (elisp) thing either.

> - subscriptions (but the user can pick subsets as the "work" and "home"
>   setup for instance) (hard)

Replace "level" by tags? But once again, that is a configuration stuff.
That should just be a Lisp list in a .gnus you just C-M-x to update.

> - the Gnus registry (medium)

Why not.

> So because this is such diverse data I think the storage backend should
> just be a key-value store.  The rtree/ELisp data merge support is really
> the only thing special about it.  And maybe it's not a Gnus-specific
> thing but could be useful for Emacs in general, in which case it should
> be called data-sync.el and the Gnus glue is gnus-sync.el.  I'm sure the
> org-mode users, for instance, would like this kind of functionality too,
> and maybe they already have it and we can just steal it :)

You're mixing 2 problems IMHO. You want to do data synchronization above
several places, and export Gnus state data so they can be reused easily
with other Gnus instances.

I think the latter is a problem you can solve, but trying to resolve the
data synchronization is not an Emacs problem. I do not want Emacs to
mess with my storage strategy nor I want you to make some cloud storage.
I mean: that's not the point.

I store my Emacs configuration in my home, in a git repository. That's
my choice. I could also store my home on a webdav exported storage, NFS,
or whatever. That's my cloud, my business. Do not try to provide
something else: I do not want you to, nor I'd like an Emacs specific
solution.

OTOH, I'd like to be able to reuse data synchronization from backends
that can store some information, like nntp (or nnrss, or anything else),
and move away things that have nothing to do in here, like for example
subscriptions or subscriptions levels. If I want to split things for
work and home, I can write it easily in Lisp.

If you want the users to change subscriptions or their levels, there is
a common interface called `cuztomize' which I think is done just for
that, and can dump Lisp code in your configuration file.

Let me write Lisp!

(Sorry if the end of the mail is a little off-topic.)

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: gnus-sync ideas

[Ted Zlatanov]

On Thu, 16 Dec 2010 11:06:04 +0100 Julien Danjou <julien@danjou.info> wrote: 

JD> On Wed, Dec 15 2010, Ted Zlatanov wrote:
>> - splitting rules (trivial but needs a view/edit interface)

JD> I do not see why it's not a configuration (elisp) thing.

I'm trying to share the configuration between Gnus installations.
Splitting rules are an essential part of the configuration.

>> - BBDB (hard but very important, and useful beyond Gnus)

JD> I do not see the point.

I take it you don't keep your BBDB under VCS and endure merge pain when
you update a record from different machines?

>> - topic hierarchy (medium)

JD> I do not see why it's not a configuration (elisp) thing either.

Because topics are an essential part of the Gnus experience, so I think
synchronizing them would make it easy to "just run Gnus" on a different
machine.

>> - subscriptions (but the user can pick subsets as the "work" and "home"
>> setup for instance) (hard)

JD> Replace "level" by tags? But once again, that is a configuration stuff.
JD> That should just be a Lisp list in a .gnus you just C-M-x to update.

Same as topics and splitting rules.

JD> You're mixing 2 problems IMHO. You want to do data synchronization above
JD> several places, and export Gnus state data so they can be reused easily
JD> with other Gnus instances.

Yes, AKA data-sync.el and gnus-sync.el.

JD> I think the latter is a problem you can solve, but trying to resolve the
JD> data synchronization is not an Emacs problem. I do not want Emacs to
JD> mess with my storage strategy nor I want you to make some cloud storage.
JD> I mean: that's not the point.

JD> I store my Emacs configuration in my home, in a git repository. That's
JD> my choice. I could also store my home on a webdav exported storage, NFS,
JD> or whatever. That's my cloud, my business. Do not try to provide
JD> something else: I do not want you to, nor I'd like an Emacs specific
JD> solution.

Sure.  But nothing has *merge* support for ELisp data structures, that's
the problem.  Thus data-sync.el will provide merge strategies.  If you
choose not to use it, you're just versioning your Gnus configuration,
which is fine.  But people want to merge marks and other data between
Gnus installations and Git will NOT do that correctly when there's a
conflict.

So basically you don't like data-sync.el.  That's cool.  You'll be able
to just use gnus-sync.el with an "overwrite-only" data-sync.el backend,
I promise.  And that backend will behave like a file (in fact it will
generate something much like newsrc.eld), and there will be much
rejoicing.

JD> If you want the users to change subscriptions or their levels, there is
JD> a common interface called `cuztomize' which I think is done just for
JD> that, and can dump Lisp code in your configuration file.

But Customize is a) sucky, b) can't handle merging, c) doesn't do large
data sets well, and d) is REALLY hard to modify at this point because so
many users and libraries depend on its exact behavior not to change.  So
I don't think it can handle what Gnus needs.

Ted

Re: gnus-sync ideas

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

JD> I do not see the point.

> I take it you don't keep your BBDB under VCS

I do.

> and endure merge pain when you update a record from different
> machines?

I do.

Basically you have to decide that one .bbdb be the master and let it
overwrite changes in the others.  Somewhat... limiting.

Re: gnus-sync ideas

[Philipp Haselwarter]

Quite some good points, Julien.

I'm using git for my emacs config, and I'd much rather be able to store
my gnus config that way. The idea of running yet-another-server just for
the gnus data really gives me the creeps. Managing stuff like
subscriptions (which are configuration and not state data) should happen
in a simple lisp file. If I subscribe to new groups I just want to be
able to just push that change with git.
The state of my imap groups should be taken care of by imap anyways.

So what's left is the state of newsgroups and maybe BBDB.


-- 
Philipp Haselwarter

Re: gnus-sync ideas

[Lars Magne Ingebrigtsen]

Philipp Haselwarter <philipp.haselwarter@gmx.de> writes:

> Managing stuff like subscriptions (which are configuration and not
> state data) should happen in a simple lisp file.

I disagree.  What groups you read, and what order they appear in the
group buffer, is state data, not configuration.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-sync ideas

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> I disagree.  What groups you read, and what order they appear in the
> group buffer, is state data, not configuration.

Agreed.

But group parameters are config, I think.

Re: gnus-sync ideas

[Ted Zlatanov]

On Wed, 15 Dec 2010 10:32:37 -0600 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> But we should generally be able to save, in order of importance
TZ> IMHO:

TZ> - marks for news-p servers (medium difficulty)

TZ> - splitting rules (trivial but needs a view/edit interface)

TZ> - BBDB (hard but very important, and useful beyond Gnus)

TZ> - topic hierarchy (medium)

TZ> - subscriptions (but the user can pick subsets as the "work" and "home"
TZ>   setup for instance) (hard)

TZ> - the Gnus registry (medium)

- scoring rules (medium)

Re: gnus-sync ideas

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

[snip!]
> First of all, how should it operate in Gnus?  IMO it should be
> immediate and not triggered as it is now; a sync should happen every
> time you enter or exit a group (and can also be triggered explicitly).

Definitely.  The way it is today, where I have to remember to stop
gnusen every time I leave them, or minimum remember to save, is a
nuisance. 

[snip!]
> Second, how should the data be stored?  IMO it should be stored in a
> free-form key-value format

Um... do you mean that the values should be blobs in elisp or somesuch
format?  (if so, what is the keys?  Not obvious to me) Or do you mean a
fine-grained "everything as a key/value pair" storage? (hm.. neither the
keys, nor the values are obvious to me here either... :-) )

> but the storage backend should be able to understand the rtree format;
> the ELisp list, vector, and plist formats; and maybe JSON and merge
> such data (with parameters that control how the merge will happen).

Hm... why so many formats?  We're not going for interoperability with
anything else (ie. why use JSON?).  And what's the difference between
elisp lists and the rtree format?  (FWIW I think the format should be
s-expressions/elisp lists, because it is simple and low cost on the
emacs side)

> I think a custom network server would actually be the best storage
> backend solution here; imap-hash.el could also work with some love;
> the file storage backend is also sensible to duplicate the current
> newsrc.eld functionality; and, of course, people will come up with a
> Git-based and RDBMS-based storage backend Because They Can.

Heh... the server could be ssh-ing to a central computer and starting an
emacs remotely to interpret the sync data, and consolidate with the lisp
list data stored on the server... "emacs in the cloud" :-)

(not serious: relying on ssh is too limiting, because it is blocked by
too many corporate firewalls)

Actually I think there only needs to be a single protocol http/https
with PUT or POST for updates (which could also return updated sync data
uploaded by other clients), and GET for initial sync on start, or
explicit syncs.

Wrt. round trip times: GET can be pipelined.  PUT and POST can't.  But
if you don't care about piggybacking updated sync info on the response
(only read upstream sync data with GET) you could just fire them off in
the background with the thread thing Lars talked about in a different
thread. 

> Third, what data are we storing?  I think marks should only be synced
> for news-p backends.  But we should generally be able to save, in order
> of importance IMHO:

> - marks for news-p servers (medium difficulty)

That I definitely want.
+1

> - splitting rules (trivial but needs a view/edit interface)

+0 (I don't use it, but don't let me stop you...:-) )

> - BBDB (hard but very important, and useful beyond Gnus)

Currently version controlled.  As you've pointed out, merge is not
really possible on the file level.
+1

> - topic hierarchy (medium)

0 (neutral, for the same reason as the next one)

> - subscriptions (but the user can pick subsets as the "work" and "home"
>   setup for instance) (hard)

-1 I like to set up those manually.  But I would like a newly subscribed
   group to get its read state from the sync server, if it is read in a
   different gnus

> - the Gnus registry (medium)

0 (neutral, because I don't use it)

- Score rules

+1 (because I use it)

> So because this is such diverse data I think the storage backend
> should just be a key-value store.  The rtree/ELisp data merge support
> is really the only thing special about it.  And maybe it's not a
> Gnus-specific thing but could be useful for Emacs in general, in which
> case it should be called data-sync.el and the Gnus glue is
> gnus-sync.el.  I'm sure the org-mode users, for instance, would like
> this kind of functionality too, and maybe they already have it and we
> can just steal it :)

Hm... I still don't have a clear idea how this maps to a key value
store.  These are nested constructs, aren't they?  Will you construct a
key by combining identificators at different level?

> Fourth, for general accessibility, data-sync.el will have a simple
> view/edit interface to load a specific key in a buffer, edit it in
> Emacs Lisp, rtree, or JSON mode, and commit it back if it validates
> (no merge, just overwrite).  This functionality could also connect
> with revisions if the storage backend supports them, so you could look
> at older versions of your data.

Why involve JSON?  Why not just lisp list structures?

> Fifth, the network server storage backend should be easy to set up and
> run.  Maybe a HTTP-backed implementation would make the most sense
> because those are so easy to set up and are not firewalled.

Yep.  It's what I think as well.  See comments above.

> Finally, what other parts of Emacs besides BBDB and org-mode could
> benefit from a general data-sync facility?

I'm sure others will think of something once it is in place.  The diary,
perhaps?  Though that has other sync possibilities (iCalendar).

Re: gnus-sync ideas

[Ted Zlatanov]

On Sun, 19 Dec 2010 21:08:21 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:

>> Second, how should the data be stored?  IMO it should be stored in a
>> free-form key-value format

SB> Um... do you mean that the values should be blobs in elisp or somesuch
SB> format?  (if so, what is the keys?  Not obvious to me) Or do you mean a
SB> fine-grained "everything as a key/value pair" storage? (hm.. neither the
SB> keys, nor the values are obvious to me here either... :-) )

The key is going to be consistent, e.g. "gnus.marks.GROUPNAME" but the
storage (data-sync.el) backend may prefix the key with something.  For a
shared networked storage backend, for instance, the prefix would be
something unique and unguessable to prevent accidental or malicious
collisions (there will also be a password).

The values will be data blobs with only one property: they can be
merged.  Thus many data formats can be supported.

>> but the storage backend should be able to understand the rtree format;
>> the ELisp list, vector, and plist formats; and maybe JSON and merge
>> such data (with parameters that control how the merge will happen).

SB> Hm... why so many formats?  We're not going for interoperability with
SB> anything else (ie. why use JSON?).  And what's the difference between
SB> elisp lists and the rtree format?  (FWIW I think the format should be
SB> s-expressions/elisp lists, because it is simple and low cost on the
SB> emacs side)

rtree is much more compact that simple sexps for marks IIUC.  So it
makes sense to use it for marks, but it's not appropriate for many other
formats.

The JSON support would be there for other uses.  I may actually convert
between JSON, rtree, and sexps on the fly but haven't decided that yet.
If I do, other clients besides ELisp-based could access the data sync
functionality.

>> I think a custom network server would actually be the best storage
>> backend solution here; imap-hash.el could also work with some love;
>> the file storage backend is also sensible to duplicate the current
>> newsrc.eld functionality; and, of course, people will come up with a
>> Git-based and RDBMS-based storage backend Because They Can.

SB> Heh... the server could be ssh-ing to a central computer and starting an
SB> emacs remotely to interpret the sync data, and consolidate with the lisp
SB> list data stored on the server... "emacs in the cloud" :-)

SB> (not serious: relying on ssh is too limiting, because it is blocked by
SB> too many corporate firewalls)

Yeah.  Also Emacs as a server is extremely limited--I thought of this.
It has to use HTTP as the transport and some kind of web service as the
RPC mechanism for a networked storage backend.  But for a local storage
backend Emacs itself can do the job just fine.

SB> Actually I think there only needs to be a single protocol http/https
SB> with PUT or POST for updates (which could also return updated sync data
SB> uploaded by other clients), and GET for initial sync on start, or
SB> explicit syncs.

SB> Wrt. round trip times: GET can be pipelined.  PUT and POST can't.  But
SB> if you don't care about piggybacking updated sync info on the response
SB> (only read upstream sync data with GET) you could just fire them off in
SB> the background with the thread thing Lars talked about in a different
SB> thread. 

That's easy to do as an optimization.  I'll try to get the basics
working first.  The HTTP GET/POST semantics should be sufficient so PUT
and DELETE won't be needed.

>> So because this is such diverse data I think the storage backend
>> should just be a key-value store.  The rtree/ELisp data merge support
>> is really the only thing special about it.  And maybe it's not a
>> Gnus-specific thing but could be useful for Emacs in general, in which
>> case it should be called data-sync.el and the Gnus glue is
>> gnus-sync.el.  I'm sure the org-mode users, for instance, would like
>> this kind of functionality too, and maybe they already have it and we
>> can just steal it :)

SB> Hm... I still don't have a clear idea how this maps to a key value
SB> store.  These are nested constructs, aren't they?  Will you construct a
SB> key by combining identificators at different level?

gnus-sync.el will do the mapping between the "Gnus state" and the
"synchronized state."  Eventually the two will converge, I hope, and
Gnus will only have a "synchronized state" which happens to be
synchronized to local disk by default.  But at first there will be a
conversion process to avoid rewriting Gnus too radically and to get the
data synchronization working properly.

Ted

Re: gnus-sync ideas

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> The values will be data blobs with only one property: they can be
> merged.  Thus many data formats can be supported.

If you have more than one type of blob, don't you need some type
information?  Or do you plan to guess the type when using the data?  Or
do you mean to normalize to a single type?

SB> Hm... why so many formats?  We're not going for interoperability with
SB> anything else (ie. why use JSON?).  And what's the difference between
SB> elisp lists and the rtree format?  (FWIW I think the format should be
SB> s-expressions/elisp lists, because it is simple and low cost on the
SB> emacs side)

> rtree is much more compact that simple sexps for marks IIUC.
> So it makes sense to use it for marks, but it's not appropriate for
> many other formats.

Umm... what do you mean by "rtree" here?  The abstract rtree data
structure?  A particular non-lisp format?  Lars' lisp implementation of
rtrees in rtree.el?

The last one is something that can be fed the lisp reader, and are (as
far as I understand s-expressions) s-expressions as good as any...?

> That's easy to do as an optimization.  I'll try to get the basics
> working first.  The HTTP GET/POST semantics should be sufficient so
> PUT and DELETE won't be needed.

Why POST and not PUT?

Nothing important.  I'm just curious as to the reasoning?  It's like
XML attributes vs. elements: why pick one over the other? (well... not
quite.  If you are using a SAX style parser there are good reasons for
picking attributes over elements if you can, which is when there is no
hierarical values to represent, and no multiple values.  With POST and
PUT there aren't really good reasons for there to be two separate
mechanisms.  My rule of thumb is to use POST if you need to transfer
multiple values, eg. the value and some metainformation about the value,
and PUT if it is just the value).

Hm... when I think about it, there is at least one piece of
metainfomation that is neccessry: a timestamp.

Re: Separating marks from other group configuration

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

On Mon, Dec 13 2010, Lars Magne Ingebrigtsen wrote:

> While making dinner here I was thinking about what Steinar said about
> separating out the non-marks stuff (group parameters and...  stuff?)
> from .newsrc.eld.  Which led me to thinking about where to store it.
> Which led me to thinking about IMAP.

As an IMAP-only user, I just want to say that the .newsrc.eld is a PITA
to me.

I'm OK that Gnus use that file to store its IMAP cache
(gnus-newsrc-alist).

But it's currently a bit more than a cache when it comes to subscription
for example. If Gnus was able to retrieve IMAP subscription + assign
group level on a configuration basis[1], I'd be happier. I'd like to not
have my mail handling depending on a data file. :)

OTOH, gnus-sync etc would be fine for people reading newsgroups and
wanting to sync their gnus-newsrs-alist on multiple accounts. :)

My 2¢,

[1] (same for topic, but I do not use them because they are stored in
    .newsrc.eld only).

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Separating marks from other group configuration

[Steinar Bang]

>>>>> Julien Danjou <julien@danjou.info>:

> OTOH, gnus-sync etc would be fine for people reading newsgroups and
> wanting to sync their gnus-newsrs-alist on multiple accounts. :)

It's what I use it for, and was a relief to get it in place.

Re: Separating marks from other group configuration

[Philipp Haselwarter]

Julien Danjou <julien@danjou.info> writes:

> On Mon, Dec 13 2010, Lars Magne Ingebrigtsen wrote:
>
>> While making dinner here I was thinking about what Steinar said about
>> separating out the non-marks stuff (group parameters and...  stuff?)
>> from .newsrc.eld.  Which led me to thinking about where to store it.
>> Which led me to thinking about IMAP.
>
> As an IMAP-only user, I just want to say that the .newsrc.eld is a PITA
> to me.

+1
If at least it weren't that messy. But especially once gnus gets
confused with the number of mails in a group, things can get quite
obscure. Like, when I want to browse my inbox, which gnus claims to
contain about 3 times the messages it actually does, "/ o 100" often just
displays the last 15 messages.
Also, I have to subscribe to new groups on every site I use gnus, making
the whole configuration rather unportable.
Maybe it'd already help to refactor the .newsrc.eld? (I know you're not
supposed to edit that manually, but sometimes it just feels like you
need to get some control over things..)

In general, adding/editing groups in gnus instead of .gnus.el does
make it quite difficult to figure out what sate your config really is
in:
For example, on my notebook I added an rss feed in the group buffer, G m
and adding all the information. Now if I want the same feed elsewhere I
have to go through all that again. I'd really love to just get some
lisp snippets that I could directly dump into my .gnus.el.


-- 
Philipp Haselwarter

Re: Separating marks from other group configuration

[Lars Magne Ingebrigtsen]

Philipp Haselwarter <philipp.haselwarter@gmx.de> writes:

> If at least it weren't that messy. But especially once gnus gets
> confused with the number of mails in a group, things can get quite
> obscure. Like, when I want to browse my inbox, which gnus claims to
> contain about 3 times the messages it actually does, "/ o 100" often
> just displays the last 15 messages.

This is a different matter -- this is an artifact from Gnus not being
told which articles actually exists in sparsely-populated groups.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Separating marks from other group configuration

[Philipp Haselwarter]

On Wed, 15 Dec 2010 20:47:06 +0100, Lars Magne Ingebrigtsen
<larsi@gnus.org> said: 

LMI> Philipp Haselwarter <philipp.haselwarter@gmx.de> writes:

>> If at least it weren't that messy. But especially once gnus gets
>> confused with the number of mails in a group, things can get quite
>> obscure. Like, when I want to browse my inbox, which gnus claims to
>> contain about 3 times the messages it actually does, "/ o 100" often
>> just displays the last 15 messages.

LMI> This is a different matter -- this is an artifact from Gnus not
LMI> being told which articles actually exists in sparsely-populated
LMI> groups.

LMI> -- (domestic pets only, the antidote for overdose, milk.)
LMI> larsi@gnus.org * Lars Magne Ingebrigtsen

Nonetheless it would be nice to have a means to recalculate the
count. When I had just like one imap account set up, I could just go and
delete the newrc.eld and it'd get recalculated the next time. But
obviously that's not a decent way to do it..

-- 
Philipp Haselwarter

Re: Separating marks from other group configuration

[Lars Magne Ingebrigtsen]

Philipp Haselwarter <philipp.haselwarter@gmx.de> writes:

> Nonetheless it would be nice to have a means to recalculate the
> count. When I had just like one imap account set up, I could just go and
> delete the newrc.eld and it'd get recalculated the next time. But
> obviously that's not a decent way to do it..

Deleting anything from the .newsrc.eld doesn't fix Gnus not knowing
which articles don't exists, really.  (Any more than `M-g' does.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Separating marks from other group configuration

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> If Gnus was able to retrieve IMAP subscription + assign group level on
> a configuration basis[1], I'd be happier.

What's IMAP subscription?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Separating marks from other group configuration

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> What's IMAP subscription?

The commands SUBSCRIBE, UNSUBSCRIBE and LSUB, apparently:
 http://tools.ietf.org/html/rfc3501#section-6.3.6
 http://tools.ietf.org/html/rfc3501#section-6.3.7
 http://tools.ietf.org/html/rfc3501#section-6.3.9
 http://tools.ietf.org/html/rfc3501#section-7.2.3

Re: Separating marks from other group configuration

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> The commands SUBSCRIBE, UNSUBSCRIBE and LSUB, apparently:
>  http://tools.ietf.org/html/rfc3501#section-6.3.6
>  http://tools.ietf.org/html/rfc3501#section-6.3.7
>  http://tools.ietf.org/html/rfc3501#section-6.3.9
>  http://tools.ietf.org/html/rfc3501#section-7.2.3

Ah, right.  Do other IMAP mail readers use these much?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Separating marks from other group configuration

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

On Wed, Dec 15 2010, Lars Magne Ingebrigtsen wrote:

>> The commands SUBSCRIBE, UNSUBSCRIBE and LSUB, apparently:
>>  http://tools.ietf.org/html/rfc3501#section-6.3.6
>>  http://tools.ietf.org/html/rfc3501#section-6.3.7
>>  http://tools.ietf.org/html/rfc3501#section-6.3.9
>>  http://tools.ietf.org/html/rfc3501#section-7.2.3
>
> Ah, right.  Do other IMAP mail readers use these much?

mutt, Thunderbird, KMail, Claws Mail, Evolution…

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Separating marks from other group configuration

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> mutt, Thunderbird, KMail, Claws Mail, Evolution…

So basically all other mail readers use LSUB instead of LIST?  Then why
does nnimap use LIST?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Separating marks from other group configuration

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 322 bytes --]

On Thu, Dec 16 2010, Lars Magne Ingebrigtsen wrote:

> So basically all other mail readers use LSUB instead of LIST?

That's configurable usually.

> Then why
> does nnimap use LIST?  :-)

Because there's no way to send {UN,}SUBSCRIBE commands via nnimap. :(

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Separating marks from other group configuration

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

>> So basically all other mail readers use LSUB instead of LIST?
>
> That's configurable usually.

What's the usual default?

>> Then why
>> does nnimap use LIST?  :-)
>
> Because there's no way to send {UN,}SUBSCRIBE commands via nnimap. :(

Well, there could be.  `C-k' in the group buffer and `u' in the browse
buffer could send un/subscribe commands, if this is useful.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Separating marks from other group configuration

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 549 bytes --]

On Thu, Dec 16 2010, Lars Magne Ingebrigtsen wrote:

>>> So basically all other mail readers use LSUB instead of LIST?
>>
>> That's configurable usually.
>
> What's the usual default?

LSUB

>>> Then why
>>> does nnimap use LIST?  :-)
>>
>> Because there's no way to send {UN,}SUBSCRIBE commands via nnimap. :(
>
> Well, there could be.  `C-k' in the group buffer and `u' in the browse
> buffer could send un/subscribe commands, if this is useful.

I think it would really be.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Separating marks from other group configuration

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

>> Well, there could be.  `C-k' in the group buffer and `u' in the browse
>> buffer could send un/subscribe commands, if this is useful.
>
> I think it would really be.

I wonder how to bootstrap this, though.  People have nnimap groups now,
and if nnimap just starts doing LSUB, the groups are going to go away,
which would be annoying.

Hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Separating marks from other group configuration

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 504 bytes --]

On Thu, Dec 16 2010, Lars Magne Ingebrigtsen wrote:

> I wonder how to bootstrap this, though.  People have nnimap groups now,
> and if nnimap just starts doing LSUB, the groups are going to go away,
> which would be annoying.

There used to be some sort of option to use LIST/LSUB in old nnimap. I'm
quite sure there was something about that, since one of my first Gnus
contribution patch was to fix a issue related to nnimap LSUB usage. :)

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Separating marks from other group configuration

[Ted Zlatanov]

On Thu, 16 Dec 2010 16:50:31 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Julien Danjou <julien@danjou.info> writes:
>> mutt, Thunderbird, KMail, Claws Mail, Evolution…

LMI> So basically all other mail readers use LSUB instead of LIST?  Then why
LMI> does nnimap use LIST?  :-)

Everyone knew but didn't want to embarass you ;)

Ted