* I can haz cloud idea @ 2012-02-16 5:03 Lars Ingebrigtsen 2012-02-16 6:49 ` Lars Ingebrigtsen ` (5 more replies) 0 siblings, 6 replies; 40+ messages in thread From: Lars Ingebrigtsen @ 2012-02-16 5:03 UTC (permalink / raw) To: ding The main problems with going all cloudy is that you have to have yet another server out there somewhere. Whether you use Dropbox/sshfs/a yet to be determined Gnus-specific server, that's not very sexy. But how about if we just cheat and put all the information on the IMAP server? That is: Gnus would create and maintain a special group on one of your IMAP servers that would contain all the data Gnus needs. You'd set this up by setting, er, something, like: (setq gnus-cloud-server '(nnimap "imap.gmail.com")) This would make Gnus save all the data on that server whenever you exit Gnus. (Or on certain intervals.) It could gzip and encrypt the data before uploading, I guess. So when you go to your other machine, you'd say `M-x gnus-cloud', and it would prompt you for the server and password(s), download the data, and then start up more or less as normal. And it would upload the data on exit, as well, so you'd be all cloudy. What would be stored on the server? Well, the .newsrc.eld file, for one. And the SCORE files, I guess. And... well, whatever else you want. It could be a simple archive "file" type, and Gnus could just decrypt, uncompress and unpack the files. Some sanity check about overwriting newer files with older files, I guess. How much data would this be? Here's my .newsrc.eld file: larsi@rusty:~/mgnus$ ls -l ~/.newsrc.eld -rw-r--r-- 1 larsi larsi 502770 2012-02-16 05:46 /home/larsi/.newsrc.eld larsi@rusty:~/mgnus$ gzip -9 -c /home/larsi/.newsrc.eld | wc 441 2222 117577 larsi@rusty:~/mgnus$ xz -c /home/larsi/.newsrc.eld | wc 258 1440 65288 A 65K download isn't too bad. Well. You'd have to base64 it before uploading, I guess. Does Emacs have xz compression built in? And built-in encryption would also be nice, but that's already planned for Emacs 24.2, I guess... -- (domestic pets only, the antidote for overdose, milk.) http://lars.ingebrigtsen.no * Sent from my Rome ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 5:03 I can haz cloud idea Lars Ingebrigtsen @ 2012-02-16 6:49 ` Lars Ingebrigtsen 2012-02-16 8:19 ` Antoine Levitt 2012-02-16 9:27 ` Steinar Bang 2012-02-16 7:00 ` Vegard Vesterheim ` (4 subsequent siblings) 5 siblings, 2 replies; 40+ messages in thread From: Lars Ingebrigtsen @ 2012-02-16 6:49 UTC (permalink / raw) To: ding If this is to work smoothly (i.e., at all), then every time you hit `g' in a Gnus, it'll have to know whether something else updated the state group. This also means that every time you exit a group or does something else that changes state, it'll have to be uploaded pretty much immediately. This also means that there would be no explicit "upload" commands. Here's how I see this working: When you first start the thing, it does a total upload. Then, every time you (say) exit a group, it only uploads the delta between the previous total upload and the current state. So after you've read a couple of groups, the delta file will look like ("gwene.com.feedburner.cheezburger" (add (read (56 . 59))) (del (tick 58))) or something. Gnus will continue to upload the entire delta file to the server on each group exit (and delete any previous deltas). Until the delta gets so big that Gnus just does a total upload again, and removes all previous data. It sucks that `g' has to query the state group, though. But it really doesn't. If we maintain a separate cloud connection, then we can just IDLE on the state group. Then every time a different Gnus somewhere does something, then the other Gnus instances can just download the state change, and update itself automatically. Totally transparently and in the background. Unless the IMAP server has dropped the connection, of course. Then `g' will have to reconnect and do the update while you wait. -- (domestic pets only, the antidote for overdose, milk.) http://lars.ingebrigtsen.no * Sent from my Rome ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 6:49 ` Lars Ingebrigtsen @ 2012-02-16 8:19 ` Antoine Levitt 2012-02-16 9:27 ` Steinar Bang 1 sibling, 0 replies; 40+ messages in thread From: Antoine Levitt @ 2012-02-16 8:19 UTC (permalink / raw) To: ding That'd be a really cute feature ! 16/02/12 07:49, Lars Ingebrigtsen > If this is to work smoothly (i.e., at all), then every time you hit `g' > in a Gnus, it'll have to know whether something else updated the state > group. This also means that every time you exit a group or does > something else that changes state, it'll have to be uploaded pretty much > immediately. This also means that there would be no explicit "upload" > commands. Please consider making the feature customizable enough to allow the possibility of an explicit sync. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 6:49 ` Lars Ingebrigtsen 2012-02-16 8:19 ` Antoine Levitt @ 2012-02-16 9:27 ` Steinar Bang 2012-02-16 9:35 ` Lars Ingebrigtsen 1 sibling, 1 reply; 40+ messages in thread From: Steinar Bang @ 2012-02-16 9:27 UTC (permalink / raw) To: ding >>>>> Lars Ingebrigtsen <larsi@gnus.org>: [snip!] > Here's how I see this working: Have you given any thought to offline/agent operations? ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 9:27 ` Steinar Bang @ 2012-02-16 9:35 ` Lars Ingebrigtsen 0 siblings, 0 replies; 40+ messages in thread From: Lars Ingebrigtsen @ 2012-02-16 9:35 UTC (permalink / raw) To: ding Steinar Bang <sb@dod.no> writes: > Have you given any thought to offline/agent operations? No... :-) But thinking about it for the length of time it's taken me to write these words, I don't think offline-ness complicates matters much. I mean, you need to go online to pull/push updates, but the computations themselves would be identical. That is, Gnus would keep a set of "baseline" data in memory. (Basically everything that is in .newsrc.eld.) Whenever Gnus does a push, it computes the difference between the baseline and the current state. If the difference is smallish, it then pushes a delta to the IMAP server. If not, it pushes the complete state. So going-online-and-then-pushing wouldn't introduce any new code. Same with pulling -- it should just work. I think. -- (domestic pets only, the antidote for overdose, milk.) http://lars.ingebrigtsen.no * Sent from my Rome ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 5:03 I can haz cloud idea Lars Ingebrigtsen 2012-02-16 6:49 ` Lars Ingebrigtsen @ 2012-02-16 7:00 ` Vegard Vesterheim 2012-02-17 17:49 ` Richard Riley 2012-02-16 8:25 ` David Engster ` (3 subsequent siblings) 5 siblings, 1 reply; 40+ messages in thread From: Vegard Vesterheim @ 2012-02-16 7:00 UTC (permalink / raw) To: ding On Thu, 16 Feb 2012 06:03:47 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: > But how about if we just cheat and put all the information on the IMAP > server? yes, Yes, YES! - Vegard V - ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 7:00 ` Vegard Vesterheim @ 2012-02-17 17:49 ` Richard Riley 2012-02-19 15:21 ` Lars Ingebrigtsen 0 siblings, 1 reply; 40+ messages in thread From: Richard Riley @ 2012-02-17 17:49 UTC (permalink / raw) To: ding Vegard Vesterheim <vegard.vesterheim@uninett.no> writes: > On Thu, 16 Feb 2012 06:03:47 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: > >> But how about if we just cheat and put all the information on the IMAP >> server? > > yes, Yes, YES! > > - Vegard V - > Use the existing gpg stuff as currently supported for auth files etc. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-17 17:49 ` Richard Riley @ 2012-02-19 15:21 ` Lars Ingebrigtsen 2012-02-19 18:04 ` Andy Moreton ` (2 more replies) 0 siblings, 3 replies; 40+ messages in thread From: Lars Ingebrigtsen @ 2012-02-19 15:21 UTC (permalink / raw) To: ding I've been gallivanting around a bit, but I've been thinking a bit more about the cloud thing. So the original idea would be to have a bundle of files distributed via IMAP, plus diffs to avoid down/uploading so much stuff. When stuff changes in the bundle, Gnus updates the appropriate files. But then I thought: This is kinda like a virtual file system. This could be implemented along similar lines like Emacs tar support. That is: We have an in-memory file system "mounted" at (say) ~/.gnus-cloud, and whenever you edit ~/.gnus-cloud/News/all.SCORE, you're really just editing the bundle kept in-memory. This means that if you start Gnus with `M-x gnus-cloud', no files would need to be written to local disk. So you'd typically have a "real" Gnus somewhere that writes stuff out to real files. All (or most) of your other Gnus instances would just rely on the data from the IMAP server. No local files! Doesn't that have a clean feeling? But then, today, I had another thought. Gnus setup consists of the .newsrc.eld file, and then a bunch of other files, like ~/.signature and ~/News/*.SCORE. But... why? Why rely on other files for the significant stuff? We control the vertical and the horizontal, so why not just rethink this? The score files, for instance, would more naturally be a structure stored in .newsrc.eld. Finding out what score files that match what group is pretty gross code-wise. Why not just make that into a nice alist and put it in the .newsrc.eld file? The same goes for other obvious stuff like ~/.signature. Just put it all in .newsrc.eld, and then distribute that via IMAP. Admittedly, I've had a few Italian beers, but it all seems to make sense if you do it that way. -- (domestic pets only, the antidote for overdose, milk.) http://lars.ingebrigtsen.no * Sent from my Rome ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-19 15:21 ` Lars Ingebrigtsen @ 2012-02-19 18:04 ` Andy Moreton 2012-02-19 22:15 ` Dan Christensen 2012-02-20 7:37 ` Lars Ingebrigtsen 2012-02-19 19:03 ` Adam Sjøgren 2012-02-21 21:23 ` Ted Zlatanov 2 siblings, 2 replies; 40+ messages in thread From: Andy Moreton @ 2012-02-19 18:04 UTC (permalink / raw) To: ding On Sun 19 Feb 2012, Lars Ingebrigtsen wrote: > I've been gallivanting around a bit, but I've been thinking a bit more > about the cloud thing. > > So the original idea would be to have a bundle of files distributed via > IMAP, plus diffs to avoid down/uploading so much stuff. When stuff > changes in the bundle, Gnus updates the appropriate files. > > But then I thought: This is kinda like a virtual file system. This > could be implemented along similar lines like Emacs tar support. That > is: We have an in-memory file system "mounted" at (say) ~/.gnus-cloud, > and whenever you edit ~/.gnus-cloud/News/all.SCORE, you're really just > editing the bundle kept in-memory. This means that if you start Gnus > with `M-x gnus-cloud', no files would need to be written to local disk. Surely this is really a lot more like a distributed version control system. After all, you do want to be able to recover the previous state after errors... So keep everything in simple self consistent local files, then use DVCS mechanisms to propagate versions. > So you'd typically have a "real" Gnus somewhere that writes stuff out to > real files. All (or most) of your other Gnus instances would just rely > on the data from the IMAP server. No local files! > > Doesn't that have a clean feeling? Shudder... sounds like "single point of failure" to me. I'd like a local copy of everything, so that when the cloudy stuff isn't working I can ignore it and carry one regardless. > But then, today, I had another thought. > > Gnus setup consists of the .newsrc.eld file, and then a bunch of other > files, like ~/.signature and ~/News/*.SCORE. But... why? > > Why rely on other files for the significant stuff? We control the > vertical and the horizontal, so why not just rethink this? > > The score files, for instance, would more naturally be a structure > stored in .newsrc.eld. Finding out what score files that match what > group is pretty gross code-wise. Why not just make that into a nice > alist and put it in the .newsrc.eld file? Please, please do *not* add more stuff to .newsrc.eld. That file is already over-complex, and trying to do too many things. Keep things simple and unixy - a single responsibility for each file. > The same goes for other obvious stuff like ~/.signature. > > Just put it all in .newsrc.eld, and then distribute that via IMAP. > > Admittedly, I've had a few Italian beers, but it all seems to make sense > if you do it that way. I hope you feel differently when you've sobered up :-) AndyM ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-19 18:04 ` Andy Moreton @ 2012-02-19 22:15 ` Dan Christensen 2012-02-20 7:41 ` Lars Ingebrigtsen 2012-02-20 7:37 ` Lars Ingebrigtsen 1 sibling, 1 reply; 40+ messages in thread From: Dan Christensen @ 2012-02-19 22:15 UTC (permalink / raw) To: ding Andy Moreton <andrewjmoreton@gmail.com> writes: > Please, please do *not* add more stuff to .newsrc.eld. I agree. > That file is already over-complex, and trying to do too many > things. Keep things simple and unixy - a single responsibility for > each file. Moreover, as soon as you have multiple clients sharing data, you are bound to get into a situation where some merging is required. (E.g. I go to work and learn that my home machine hasn't synced; I've got to get work done, so I've got to make changes at work as well.) Merging is generally easier when things are stored in logically separate containers. In addition, updating single "records" should be fast, and if everything is stored in the .newsrc.eld on an abstract filesystem, that's hard, and you have to resort to storing diffs, etc, to be efficient. Even more, it's doubtful that I'm going to want to sync the full .newsrc.eld, since on some machines I'll access different servers. Sync/cloud should be configured separately for each backend, if only for efficiency. I'm also not sure that a DCVS is a magic bullet here, since we're going to need problem specific merging and updating. For example, Gnus should remember the last state it saw on the server, and propagate changes back and forth, rather than copying over the new data (in case it has also been changed by another client). I haven't looked into it closely, but doesn't Ted's gnus-sync framework seem like a good basis for continued development? This really seems to me to fit into the database point of view. I also have to say that I always liked the idea of having the backends store marks. It allows naive methods of syncing and backup to just work, and it makes the backends behave more like nnimap which is now becoming one of the most used backends. I feels to me like the syncing gnus already has to do between it's knowledge of the marks in an nnimap group and the remote server's knowledge is really similar to what needs to happen for other backends in a cloud set-up. This is also closely related to the plugged/unplugged/agent feature. Can all of this be merged into one concept? 0) Gnus doesn't store any marks in the .newsrc.eld. 1) On startup, Gnus asks the backend for the marks, and displays the group. 2) As the user changes things, Gnus keeps track of the changes made to marks. 3) As various intervals, Gnus syncs the changes to the backend. Being "unplugged" which just mean deferring any syncs. The user could configure the backend to be nnimap, which already handles storing the marks on a server. Or the user could configure the backend to be nnml, with marks just stored locally. Or the user could configure the backend to be nnml, with marks stored on a server (e.g. Ted's LeSync idea, or some other remote storage). It really seems like getting all the marks out of .newsrc.eld and putting them in the backends would unify many ideas in Gnus. (And by default, servers could just store marks in a file like .newsrc.server-name.eld, using nnoo common code, so there wouldn't need to be code duplication in the backends. Even the default syncing code could be in common, but overridden by servers like nnimap that have a custom way to store marks.) Dan ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-19 22:15 ` Dan Christensen @ 2012-02-20 7:41 ` Lars Ingebrigtsen 2012-02-20 19:20 ` Dan Christensen 0 siblings, 1 reply; 40+ messages in thread From: Lars Ingebrigtsen @ 2012-02-20 7:41 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > I'm also not sure that a DCVS is a magic bullet here, since we're > going to need problem specific merging and updating. For example, > Gnus should remember the last state it saw on the server, and > propagate changes back and forth, rather than copying over the > new data (in case it has also been changed by another client). Gnus will be uploading diffs, of course, and not the complete .newsrc.eld file. > It really seems like getting all the marks out of .newsrc.eld > and putting them in the backends would unify many ideas in Gnus. And going in the other direction, which Ma Gnus is, also unifies many things. :-) As for the "merging different views" -- that's not going to happen in any meaningful way. If you're reading stuff offline on two different machines (without ever going online), one client is going to win when you're finally going online. -- (domestic pets only, the antidote for overdose, milk.) http://lars.ingebrigtsen.no * Sent from my Rome ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-20 7:41 ` Lars Ingebrigtsen @ 2012-02-20 19:20 ` Dan Christensen 2012-03-10 1:18 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 40+ messages in thread From: Dan Christensen @ 2012-02-20 19:20 UTC (permalink / raw) To: ding I really think syncing an entire .newsrc.eld file is the wrong approach. On my home machine, I read lots of nntp groups, which I never want to read on other machines, so syncing that data is inefficient and risks data loss if the remote machine updates the .newsrc.eld after I've done some reading on the local machine. Similarly, I don't want my imap groups synced, since imap already handles that. Syncing this too would be inefficient and risk data loss. This definitely should be configured on a server by server basis. Lars Ingebrigtsen <larsi@gnus.org> writes: > As for the "merging different views" -- that's not going to happen in > any meaningful way. If you're reading stuff offline on two different > machines (without ever going online), one client is going to win when > you're finally going online. If instead of diffs you upload actions like "(mark-read 12 (34 . 37) 44)" then you get lots of advantages: - very small amount of data to upload: a diff would include the old data, all the marks that didn't change, and context lines - diffs are fragile, as they might not apply cleanly if things get out of sync - this method automatically handles merging to some degree, as instead of uploading all marks for a group it only sends the changes When Gnus wants to get the marks from the server, it'll download a baseline list of marks and all of the actions, and apply them dynamically. Eventually the list of actions will get long and the baseline marks will get rewritten, but this could happen infrequently enough that the risks of overwriting other data is small. But I still think using Ted's approach might be better, especially since I think he offered to host a server. Dan ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-20 19:20 ` Dan Christensen @ 2012-03-10 1:18 ` Lars Magne Ingebrigtsen 0 siblings, 0 replies; 40+ messages in thread From: Lars Magne Ingebrigtsen @ 2012-03-10 1:18 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > On my home machine, I read lots of nntp groups, which I never want to > read on other machines, so syncing that data is inefficient and risks > data loss if the remote machine updates the .newsrc.eld after I've > done some reading on the local machine. And the same goes for nnml/nnfolder groups, which would be useless to sync. So some sort of control would be necessary here to determine what parts of the .newsrc.eld data to share. > Similarly, I don't want my imap groups synced, since imap already > handles that. Syncing this too would be inefficient and risk data loss. Well, yes and no. The .newsrc.eld file keeps track of sequence numbers (in the QRESYNC case) and max seen articles (in the non-QRESYNC) case. Syncing with an IMAP server involves a lot of network traffic, and the longer since you've synced, the more data has to be transferred. There's a large data gain to be had by syncing the .newsrc.eld data for nnimap groups too, I think. >> As for the "merging different views" -- that's not going to happen in >> any meaningful way. If you're reading stuff offline on two different >> machines (without ever going online), one client is going to win when >> you're finally going online. > > If instead of diffs you upload actions like "(mark-read 12 (34 . 37) 44)" > then you get lots of advantages: > > - very small amount of data to upload: a diff would include the old > data, all the marks that didn't change, and context lines > - diffs are fragile, as they might not apply cleanly if things get > out of sync > - this method automatically handles merging to some degree, as instead > of uploading all marks for a group it only sends the changes Yes, I think that's what I meant by "diffs". :-) But what I meant was that if you mark an article as unread in one Gnus, and then read in another, without going online, one of them is going to win when you finally go online. For that particular article. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog http://lars.ingebrigtsen.no/ ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-19 18:04 ` Andy Moreton 2012-02-19 22:15 ` Dan Christensen @ 2012-02-20 7:37 ` Lars Ingebrigtsen 2012-02-20 7:51 ` Lars Ingebrigtsen 1 sibling, 1 reply; 40+ messages in thread From: Lars Ingebrigtsen @ 2012-02-20 7:37 UTC (permalink / raw) To: Andy Moreton; +Cc: ding Andy Moreton <andrewjmoreton@gmail.com> writes: > So keep everything in simple self consistent local files, then use DVCS > mechanisms to propagate versions. That doesn't sound very user-friendly. > I hope you feel differently when you've sobered up :-) Nope. :-) -- (domestic pets only, the antidote for overdose, milk.) http://lars.ingebrigtsen.no * Sent from my Rome ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-20 7:37 ` Lars Ingebrigtsen @ 2012-02-20 7:51 ` Lars Ingebrigtsen 0 siblings, 0 replies; 40+ messages in thread From: Lars Ingebrigtsen @ 2012-02-20 7:51 UTC (permalink / raw) To: ding As an example of stuff that should go into the .newsrc.eld file is user settings like topic mode. If the user taps `t', then topic mode should be switched on the next time Gnus starts up without the user having to put (add-hook 'gnus-group-mode-hook 'gnus-topic-mode) into .gnus.el, which is just absurd. So there will be a new section in the .newsrc.eld file for "saved state" kind of things: (setq gnus-saved-state '((topics t) (offline nil))) -- (domestic pets only, the antidote for overdose, milk.) http://lars.ingebrigtsen.no * Sent from my Rome ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-19 15:21 ` Lars Ingebrigtsen 2012-02-19 18:04 ` Andy Moreton @ 2012-02-19 19:03 ` Adam Sjøgren 2012-02-21 21:23 ` Ted Zlatanov 2 siblings, 0 replies; 40+ messages in thread From: Adam Sjøgren @ 2012-02-19 19:03 UTC (permalink / raw) To: ding On Sun, 19 Feb 2012 16:21:51 +0100, Lars wrote: > The same goes for other obvious stuff like ~/.signature. I like to generate my .signature using a script that writes a file; nice and simple. (Surely a detail; just sayin'...) Best regards, Adam -- "My internal clock is on Tokyo time." Adam Sjøgren asjo@koldfront.dk ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-19 15:21 ` Lars Ingebrigtsen 2012-02-19 18:04 ` Andy Moreton 2012-02-19 19:03 ` Adam Sjøgren @ 2012-02-21 21:23 ` Ted Zlatanov 2012-03-10 1:12 ` Lars Magne Ingebrigtsen 2 siblings, 1 reply; 40+ messages in thread From: Ted Zlatanov @ 2012-02-21 21:23 UTC (permalink / raw) To: ding On Sun, 19 Feb 2012 16:21:51 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: LI> So the original idea would be to have a bundle of files distributed via LI> IMAP, plus diffs to avoid down/uploading so much stuff. When stuff LI> changes in the bundle, Gnus updates the appropriate files. The actual amount of data for a group, as a JSON object in LeSync, is tiny. I would avoid deltas if possible. They are fragile, hard to debug, and unshareable with other MUAs. Put all the data a group needs in a single place (a message in the mailbox, in this case, or a CouchDB document in LeSync) and rewrite the whole document atomically when it changes. Yes, you have a versioning problem, if someone else writes before you. So use a revision number. Please look at how CouchDB does it: only save the document if the latest document's version matches the latest version you know. Otherwise, reconcile with the latest saved. LI> Why rely on other files for the significant stuff? We control the LI> vertical and the horizontal, so why not just rethink this? LI> The score files, for instance, would more naturally be a structure LI> stored in .newsrc.eld. Finding out what score files that match what LI> group is pretty gross code-wise. Why not just make that into a nice LI> alist and put it in the .newsrc.eld file? LI> The same goes for other obvious stuff like ~/.signature. LI> Just put it all in .newsrc.eld, and then distribute that via IMAP. I would have the gnus-sync backend store user profiles, like the posting styles in Gnus. The profile would point to or inline the signature file, score files, SMTP server, and everything else the user needs, as customizations. Then you pick a profile on each machine you use. I would make each profile, with all the customizations, a single document that users can edit and understand. Don't spread things out just because you can. LI> Ted Zlatanov <tzz@lifelogs.com> writes: >> Why not add a gnus-sync backend to do this? LI> Because I want it all to be asynchronous, opportunistic and streaming. That's fine. The gnus-sync backend should support that when you call `gnus-sync-save'. You're taking the mechanics, the way code will operate, and using that as an argument against the architecture, the way data and user-customizable variables are planned. With CouchDB you can stream by opening a persistent connection to the server. url.el can be asynchronous, and so can you against CouchDB. I'm not sure what "opportunistic" means exactly, but if you mean "save when possible" then I think that would work. Just realize that by setting the gnus-sync backend to files, you can be backwards compatible with the current Gnus. That way you don't have to have a special "cloud mode" but simply "sync to local filesystem." Your way is a special case all the way from data layout to mechanics. LI> When exiting a group, we'll be squirting off an APPEND command to the LI> IMAP server (with the current delta), but we don't wait for it to LI> return finish. We queue them up, and if we get three group exits before LI> the first APPEND has finished, we just skip the second one. LI> Conversely, using IDLE on the mail box will keep the other instances LI> updated asynchronously without anybody having to issue any commands. LI> It'll be like using the IMAP synchronisation methodology, only for all LI> backends and stuff. OK. But that's nearly impossible to share with other MUAs, unlike my LeSync proposal, and I hope you don't use deltas as I said above. >> The CouchDB gnus-sync backend has built-in document versioning, which I >> think is really nice. LI> I don't think having CouchDB as a prerequisite is totally reasonable. LI> It should just work without having to install anything. You're basing your side on a working, always-reachable IMAP server, which is less reasonable in the context of the Internet and requires significant infrastructure. Port 80 is much more likely to be open than other ports, especially in a corporate environment. I didn't decide on CouchDB because it's trendy (it's SO 2009!), but because it's a schemaless database+web server hybrid with custom views (e.g. I can put together a XML view of the marks that NewsTap can import directly). You can get a free instance at http://iriscouch.com or https://cloudant.com and get running in minutes, or set your own up even quicker, or I can set one up for you at WHATEVER.lesync.info. Ted ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-21 21:23 ` Ted Zlatanov @ 2012-03-10 1:12 ` Lars Magne Ingebrigtsen 2012-03-10 12:43 ` Reiner Steib 2012-03-10 13:06 ` Ted Zlatanov 0 siblings, 2 replies; 40+ messages in thread From: Lars Magne Ingebrigtsen @ 2012-03-10 1:12 UTC (permalink / raw) To: ding Ted Zlatanov <tzz@lifelogs.com> writes: > LI> It'll be like using the IMAP synchronisation methodology, only for all > LI> backends and stuff. > > OK. But that's nearly impossible to share with other MUAs, unlike my > LeSync proposal, and I hope you don't use deltas as I said above. I don't think sharing this data with other MUAs should be a primary design goal here... > You're basing your side on a working, always-reachable IMAP server, > which is less reasonable in the context of the Internet and requires > significant infrastructure. If you can't reach the IMAP server, you can't read your email, which makes the whole thing somewhat moot. > Port 80 is much more likely to be open than other ports, especially in > a corporate environment. The use case here (which I think is the typical use case, these days) is "I read email all the time, and then I read a couple of groups from eternal-news, and 74 groups from news.gmane.org, and I want those extra groups to sync just like my mail". Like magic, without having to know or set up anything else. > I didn't decide on CouchDB because it's trendy (it's SO 2009!), But it has web scale! Or was that MongoDB? -- (domestic pets only, the antidote for overdose, milk.) bloggy blog http://lars.ingebrigtsen.no/ ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-03-10 1:12 ` Lars Magne Ingebrigtsen @ 2012-03-10 12:43 ` Reiner Steib 2012-03-10 13:06 ` Ted Zlatanov 1 sibling, 0 replies; 40+ messages in thread From: Reiner Steib @ 2012-03-10 12:43 UTC (permalink / raw) To: ding On Sat, Mar 10 2012, Lars Magne Ingebrigtsen wrote: > But it has web scale! > > Or was that MongoDB? http://www.mongodb-is-web-scale.com/ ;-) Bye, Reiner. -- ,,, (o o) ---ooO-(_)-Ooo--- | PGP key available | http://rsteib.home.pages.de/ ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-03-10 1:12 ` Lars Magne Ingebrigtsen 2012-03-10 12:43 ` Reiner Steib @ 2012-03-10 13:06 ` Ted Zlatanov 2012-07-18 14:31 ` Ted Zlatanov 1 sibling, 1 reply; 40+ messages in thread From: Ted Zlatanov @ 2012-03-10 13:06 UTC (permalink / raw) To: Lars Magne Ingebrigtsen; +Cc: ding On Sat, 10 Mar 2012 02:12:29 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: LMI> Ted Zlatanov <tzz@lifelogs.com> writes: LI> It'll be like using the IMAP synchronisation methodology, only for all LI> backends and stuff. >> >> OK. But that's nearly impossible to share with other MUAs, unlike my >> LeSync proposal, and I hope you don't use deltas as I said above. LMI> I don't think sharing this data with other MUAs should be a primary LMI> design goal here... LMI> The use case here (which I think is the typical use case, these days) is LMI> "I read email all the time, and then I read a couple of groups from LMI> eternal-news, and 74 groups from news.gmane.org, and I want those extra LMI> groups to sync just like my mail". Like magic, without having to know LMI> or set up anything else. OK, you're solving a different problem. Mine is "I want to read and sync NNTP groups from various MUAs." I'll keep maintaining gnus-sync.el separately from you, so I can keep it open to other MUAs. Ted ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-03-10 13:06 ` Ted Zlatanov @ 2012-07-18 14:31 ` Ted Zlatanov 2012-07-18 20:52 ` Steinar Bang 0 siblings, 1 reply; 40+ messages in thread From: Ted Zlatanov @ 2012-07-18 14:31 UTC (permalink / raw) To: Lars Magne Ingebrigtsen; +Cc: ding On Sat, 10 Mar 2012 08:06:13 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: TZ> On Sat, 10 Mar 2012 02:12:29 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: LMI> Ted Zlatanov <tzz@lifelogs.com> writes: LI> It'll be like using the IMAP synchronisation methodology, only for all LI> backends and stuff. >>> >>> OK. But that's nearly impossible to share with other MUAs, unlike my >>> LeSync proposal, and I hope you don't use deltas as I said above. LMI> I don't think sharing this data with other MUAs should be a primary LMI> design goal here... LMI> The use case here (which I think is the typical use case, these days) is LMI> "I read email all the time, and then I read a couple of groups from LMI> eternal-news, and 74 groups from news.gmane.org, and I want those extra LMI> groups to sync just like my mail". Like magic, without having to know LMI> or set up anything else. TZ> OK, you're solving a different problem. Mine is "I want to read and TZ> sync NNTP groups from various MUAs." TZ> I'll keep maintaining gnus-sync.el separately from you, so I can keep it TZ> open to other MUAs. Due to external commitments I'm not working on this for a while, but wanted to write a followup so people can comment. I think general "cloudiness" is a good thing nowadays, but sacrificing Lisp's power to be compatible with less powerful MUAs is not. So a Lispy cloudiness is a better goal, perhaps implemented with an IMAP backend. All of the Emacs ecosystem could benefit from such functionality: BBDB, general data storage, auth-sources, .emacs and installed ELPA packages, and so on. But even just for Gnus it's a good thing. I've been using CouchDB a lot with gnus-sync.el, and think you were right about relying on CouchDB itself. It's a pain in the long term, like any database. It's not Lispy at all, either. I'd like to work with you set up an IMAP sync backend, perhaps driven by gnus-sync.el, and rework gnus-sync.el as needed. The file backend should stay if possible; the LeSync protocol should simply become a Lisp data merge (we'll give the users the tools to upgrade). The main reason I agree is that I think the Lisp to JSON and back conversion is not good. Lisp data is so much more capable than JSON, that flattening it into JSON makes sync very complicated. Also I think there's no benefit to sharing that data with other MUAs, because the standalone MUA is a dying breed. Web-based MUAs are the norm now, and they don't share well... unless you use IMAP, in which case Gnus works fine already. For NNTP marks, the need for sharing is just not there. Ted ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-07-18 14:31 ` Ted Zlatanov @ 2012-07-18 20:52 ` Steinar Bang 0 siblings, 0 replies; 40+ messages in thread From: Steinar Bang @ 2012-07-18 20:52 UTC (permalink / raw) To: ding >>>>> Ted Zlatanov <tzz@lifelogs.com>: > Due to external commitments I'm not working on this for a while, but > wanted to write a followup so people can comment. I think general > "cloudiness" is a good thing nowadays, but sacrificing Lisp's power to > be compatible with less powerful MUAs is not. I agree. > So a Lispy cloudiness is a better goal, perhaps implemented with an > IMAP backend. I agree. > All of the Emacs ecosystem could benefit from such functionality: > BBDB, general data storage, auth-sources, .emacs and installed ELPA > packages, and so on. But even just for Gnus it's a good thing. I agree (again...). > I've been using CouchDB a lot with gnus-sync.el, and think you were > right about relying on CouchDB itself. It's a pain in the long term, > like any database. Yep. The current problem I have (when traveling) is getting auth to work, so I don't have to do "Allow from" subnets. > It's not Lispy at all, either. Nope. > I'd like to work with you set up an IMAP sync backend, perhaps driven by > gnus-sync.el, and rework gnus-sync.el as needed. The file backend > should stay if possible; the LeSync protocol should simply become a Lisp > data merge (we'll give the users the tools to upgrade). I like this. :-) I'm volunteering to try this out as an early adopter...:-) ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 5:03 I can haz cloud idea Lars Ingebrigtsen 2012-02-16 6:49 ` Lars Ingebrigtsen 2012-02-16 7:00 ` Vegard Vesterheim @ 2012-02-16 8:25 ` David Engster 2012-02-16 9:12 ` Lars Ingebrigtsen ` (3 more replies) 2012-02-16 13:07 ` Ted Zlatanov ` (2 subsequent siblings) 5 siblings, 4 replies; 40+ messages in thread From: David Engster @ 2012-02-16 8:25 UTC (permalink / raw) To: ding Lars Ingebrigtsen writes: > But how about if we just cheat and put all the information on the IMAP > server? I think it's a good idea, but I'd like it even more if this could be somehow hooked into some existing framework for storing data remotely, and that framework in Emacs is Tramp. Ted coded IMAP support for Tramp a while ago, but it was removed again for reasons I don't know. -David ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 8:25 ` David Engster @ 2012-02-16 9:12 ` Lars Ingebrigtsen 2012-02-16 9:23 ` David Engster 2012-02-16 9:25 ` Steinar Bang ` (2 subsequent siblings) 3 siblings, 1 reply; 40+ messages in thread From: Lars Ingebrigtsen @ 2012-02-16 9:12 UTC (permalink / raw) To: ding David Engster <deng@randomsample.de> writes: > I think it's a good idea, but I'd like it even more if this could be > somehow hooked into some existing framework for storing data remotely, > and that framework in Emacs is Tramp. I don't really see the point of using Tramp. nnimap is perfectly capable of up/downloading stuff on its own. And this has to be done with a high degree of asynchronocity and waiting on IDLE, etc, so Tramp would just get in the way. -- (domestic pets only, the antidote for overdose, milk.) http://lars.ingebrigtsen.no * Sent from my Rome ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 9:12 ` Lars Ingebrigtsen @ 2012-02-16 9:23 ` David Engster 2012-02-16 9:29 ` Steinar Bang 0 siblings, 1 reply; 40+ messages in thread From: David Engster @ 2012-02-16 9:23 UTC (permalink / raw) To: ding Lars Ingebrigtsen writes: > David Engster <deng@randomsample.de> writes: > >> I think it's a good idea, but I'd like it even more if this could be >> somehow hooked into some existing framework for storing data remotely, >> and that framework in Emacs is Tramp. > > I don't really see the point of using Tramp. It would make things configurable. People could easily switch between imap, ssh, rsync, etc. for syncing their Gnus stuff. > nnimap is perfectly capable of up/downloading stuff on its own. And > this has to be done with a high degree of asynchronocity and waiting > on IDLE, etc, so Tramp would just get in the way. Yes. In any case, I think IMAP is the way to go; configurability is nice'n'all, but if it is too much pain, just use plain nnimap. -David ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 9:23 ` David Engster @ 2012-02-16 9:29 ` Steinar Bang 0 siblings, 0 replies; 40+ messages in thread From: Steinar Bang @ 2012-02-16 9:29 UTC (permalink / raw) To: ding >>>>> David Engster <deng@randomsample.de>: > It would make things configurable. People could easily switch between > imap, ssh, rsync, etc. for syncing their Gnus stuff. Speaking as the current gnus-sync user base: this isn't configurability that the user base wants, or needs. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 8:25 ` David Engster 2012-02-16 9:12 ` Lars Ingebrigtsen @ 2012-02-16 9:25 ` Steinar Bang 2012-02-16 11:55 ` Greg Troxel 2012-02-16 12:51 ` Ted Zlatanov 3 siblings, 0 replies; 40+ messages in thread From: Steinar Bang @ 2012-02-16 9:25 UTC (permalink / raw) To: ding >>>>> David Engster <deng@randomsample.de>: > I think it's a good idea, but I'd like it even more if this could be > somehow hooked into some existing framework for storing data remotely, > and that framework in Emacs is Tramp. The first version of gnus-sync used tramp, and I used an SSH URL with tramp to do the syncing. Tramp was slooooow. The current gnus-sync, which uses an HTTP RESTful protocol (the CouchDB protocol), is lightning fast in comparison. > Ted coded IMAP support for Tramp a while ago, but it was removed again > for reasons I don't know. I think it was because it had many issues and few users...? ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 8:25 ` David Engster 2012-02-16 9:12 ` Lars Ingebrigtsen 2012-02-16 9:25 ` Steinar Bang @ 2012-02-16 11:55 ` Greg Troxel 2012-02-16 12:23 ` David Engster 2012-02-16 12:54 ` Ted Zlatanov 2012-02-16 12:51 ` Ted Zlatanov 3 siblings, 2 replies; 40+ messages in thread From: Greg Troxel @ 2012-02-16 11:55 UTC (permalink / raw) To: ding [-- Attachment #1: Type: text/plain, Size: 945 bytes --] David Engster <deng@randomsample.de> writes: > Lars Ingebrigtsen writes: >> But how about if we just cheat and put all the information on the IMAP >> server? > > I think it's a good idea, but I'd like it even more if this could be > somehow hooked into some existing framework for storing data remotely, > and that framework in Emacs is Tramp. > > Ted coded IMAP support for Tramp a while ago, but it was removed again > for reasons I don't know. I thin the key question here is Is what gnus-sync needs just a network filesystem, or something special? If it works with a filesystem, then one can use all sorts of things, depending on what makes sense. Some people might want sync but not have an IMAP account (but then they probably have a news-only use case). But IMAP seems like a good place, as many people will already have it set up. I think the hard part is the conflict/merge model, for use with clients in different places. [-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 11:55 ` Greg Troxel @ 2012-02-16 12:23 ` David Engster 2012-02-16 12:54 ` Ted Zlatanov 1 sibling, 0 replies; 40+ messages in thread From: David Engster @ 2012-02-16 12:23 UTC (permalink / raw) To: Greg Troxel; +Cc: ding Greg Troxel writes: > David Engster <deng@randomsample.de> writes: > >> Lars Ingebrigtsen writes: >>> But how about if we just cheat and put all the information on the IMAP >>> server? >> >> I think it's a good idea, but I'd like it even more if this could be >> somehow hooked into some existing framework for storing data remotely, >> and that framework in Emacs is Tramp. >> >> Ted coded IMAP support for Tramp a while ago, but it was removed again >> for reasons I don't know. > > I thin the key question here is > > Is what gnus-sync needs just a network filesystem, or something > special? Yes, it's probably leaning toward 'something special'. What I like about the file-based approach is that you can easily do backups. Having nightly versions of my .newsrc has saved me hours of work several times already. -David ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 11:55 ` Greg Troxel 2012-02-16 12:23 ` David Engster @ 2012-02-16 12:54 ` Ted Zlatanov 1 sibling, 0 replies; 40+ messages in thread From: Ted Zlatanov @ 2012-02-16 12:54 UTC (permalink / raw) To: ding On Thu, 16 Feb 2012 06:55:53 -0500 Greg Troxel <gdt@work.lexort.com> wrote: GT> I thin the key question here is GT> Is what gnus-sync needs just a network filesystem, or something GT> special? GT> If it works with a filesystem, then one can use all sorts of things, GT> depending on what makes sense. Some people might want sync but not have GT> an IMAP account (but then they probably have a news-only use case). But GT> IMAP seems like a good place, as many people will already have it set GT> up. GT> I think the hard part is the conflict/merge model, for use with clients GT> in different places. gnus-sync.el works against backends. The idea is to be flexible, so it should work against any storage. CouchDB happens to be the best schemaless document storage available today for our purposes, especially because of the built-in REST methods but also because JSON is a good portable format. Ted ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 8:25 ` David Engster ` (2 preceding siblings ...) 2012-02-16 11:55 ` Greg Troxel @ 2012-02-16 12:51 ` Ted Zlatanov 3 siblings, 0 replies; 40+ messages in thread From: Ted Zlatanov @ 2012-02-16 12:51 UTC (permalink / raw) To: ding On Thu, 16 Feb 2012 09:25:45 +0100 David Engster <deng@randomsample.de> wrote: DE> Ted coded IMAP support for Tramp a while ago, but it was removed again DE> for reasons I don't know. There were no users of that code, so I didn't think it worth the effort. Ted ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 5:03 I can haz cloud idea Lars Ingebrigtsen ` (2 preceding siblings ...) 2012-02-16 8:25 ` David Engster @ 2012-02-16 13:07 ` Ted Zlatanov 2012-02-20 7:48 ` Lars Ingebrigtsen 2012-03-03 3:20 ` TSUCHIYA Masatoshi 2014-10-15 23:31 ` TSUCHIYA Masatoshi 5 siblings, 1 reply; 40+ messages in thread From: Ted Zlatanov @ 2012-02-16 13:07 UTC (permalink / raw) To: ding On Thu, 16 Feb 2012 06:03:47 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: LI> The main problems with going all cloudy is that you have to have yet LI> another server out there somewhere. Whether you use Dropbox/sshfs/a yet LI> to be determined Gnus-specific server, that's not very sexy. LI> But how about if we just cheat and put all the information on the IMAP LI> server? LI> That is: Gnus would create and maintain a special group on one of your LI> IMAP servers that would contain all the data Gnus needs. LI> You'd set this up by setting, er, something, like: LI> (setq gnus-cloud-server '(nnimap "imap.gmail.com")) Why not add a gnus-sync backend to do this? LI> This would make Gnus save all the data on that server whenever you exit LI> Gnus. (Or on certain intervals.) It could gzip and encrypt the data LI> before uploading, I guess. LI> So when you go to your other machine, you'd say `M-x gnus-cloud', and it LI> would prompt you for the server and password(s), download the data, and LI> then start up more or less as normal. LI> And it would upload the data on exit, as well, so you'd be all LI> cloudy. This is what `gnus-sync-read' and `gnus-sync-save' do already, for some of the newsrc data. There's no problem with storing more, my only issue was to figure out how to synchronize it all cleanly. I also want to use gnus-sync to store the gnus-registry and even BBDB data. So it makes sense to make this a general facility. LI> What would be stored on the server? Well, the .newsrc.eld file, for LI> one. And the SCORE files, I guess. And... well, whatever else you LI> want. It could be a simple archive "file" type, and Gnus could just LI> decrypt, uncompress and unpack the files. Some sanity check about LI> overwriting newer files with older files, I guess. The CouchDB gnus-sync backend has built-in document versioning, which I think is really nice. Also it's portable through JSON and HTTP (what I call the LeSync protocol), so other newsreaders can support it too. The developer of NewsTap, a iOS app, is on board with this and told me it's on top of his feature list. LI> If this is to work smoothly (i.e., at all), then every time you hit `g' LI> in a Gnus, it'll have to know whether something else updated the state LI> group. This also means that every time you exit a group or does LI> something else that changes state, it'll have to be uploaded pretty much LI> immediately. This also means that there would be no explicit "upload" LI> commands. So basically a `gnus-sync-save' on group exit. I would amortize it so it's not done so frequently (currently, gnus-sync saves when the newsrc is saved) because network transactions are expensive. LI> Here's how I see this working: LI> When you first start the thing, it does a total upload. Then, every LI> time you (say) exit a group, it only uploads the delta between the LI> previous total upload and the current state. So after you've read a LI> couple of groups, the delta file will look like LI> ("gwene.com.feedburner.cheezburger" (add (read (56 . 59))) (del (tick 58))) LI> or something. Gnus will continue to upload the entire delta file to the LI> server on each group exit (and delete any previous deltas). Until the LI> delta gets so big that Gnus just does a total upload again, and removes LI> all previous data. LI> It sucks that `g' has to query the state group, though. But it really LI> doesn't. If we maintain a separate cloud connection, then we can just LI> IDLE on the state group. Then every time a different Gnus somewhere LI> does something, then the other Gnus instances can just download the LI> state change, and update itself automatically. Totally transparently LI> and in the background. LI> Unless the IMAP server has dropped the connection, of course. Then `g' LI> will have to reconnect and do the update while you wait. With CouchDB, at least, you can do a differential poll: tell me what has changed since change X (integer numeric sequence). You can also open a persistent connection and get changes live when they are made. So this could be, again, a gnus-sync feature per backend, and the gnus-sync LeSync and IMAP backends would support live updates while the file backend wouldn't. CouchDB's document model is interesting: you have to read the current revision to write a new one back. So you can't accidentally overwrite a document, you have to read it first (which gives you a chance to merge). So at least for CouchDB/LeSync you don't need to do delta files. Ted ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 13:07 ` Ted Zlatanov @ 2012-02-20 7:48 ` Lars Ingebrigtsen 2012-02-25 9:46 ` Steinar Bang 0 siblings, 1 reply; 40+ messages in thread From: Lars Ingebrigtsen @ 2012-02-20 7:48 UTC (permalink / raw) To: ding Ted Zlatanov <tzz@lifelogs.com> writes: > Why not add a gnus-sync backend to do this? Because I want it all to be asynchronous, opportunistic and streaming. When exiting a group, we'll be squirting off an APPEND command to the IMAP server (with the current delta), but we don't wait for it to return finish. We queue them up, and if we get three group exits before the first APPEND has finished, we just skip the second one. Conversely, using IDLE on the mail box will keep the other instances updated asynchronously without anybody having to issue any commands. It'll be like using the IMAP synchronisation methodology, only for all backends and stuff. > The CouchDB gnus-sync backend has built-in document versioning, which I > think is really nice. I don't think having CouchDB as a prerequisite is totally reasonable. It should just work without having to install anything. -- (domestic pets only, the antidote for overdose, milk.) http://lars.ingebrigtsen.no * Sent from my Rome ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-20 7:48 ` Lars Ingebrigtsen @ 2012-02-25 9:46 ` Steinar Bang 2012-03-10 1:07 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 40+ messages in thread From: Steinar Bang @ 2012-02-25 9:46 UTC (permalink / raw) To: ding >>>>> Lars Ingebrigtsen <larsi@gnus.org>: > I don't think having CouchDB as a prerequisite is totally reasonable. > It should just work without having to install anything. Even though I have set up CouchDB myself, I tend to agree: one extra server, one separate type of connection that has to work, one extra set of authentication info that has to be supplied I think relying on having an IMAP server available, for this to work, is reasonable. I think writing something as close to the metal of nnimap will boost performance. I have tried running gnus-sync with tramp as the transport mechanism, and that was slow (tramp over ssh. I never got around to try tramp over IMAP). I also think that sending the full .newsrc.eld in gzipped form is something that should be explored. My .newsrc.eld on this netbook is 780kB, which isn't big by today's network usage standards. Using "gzip --best" brings it down to 140kB, which is... nothing, really. (of course sending the entire .newsrc.eld in gzipped form would break the opportunistic streaming bit...) ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-25 9:46 ` Steinar Bang @ 2012-03-10 1:07 ` Lars Magne Ingebrigtsen 0 siblings, 0 replies; 40+ messages in thread From: Lars Magne Ingebrigtsen @ 2012-03-10 1:07 UTC (permalink / raw) To: ding Steinar Bang <sb@dod.no> writes: > I also think that sending the full .newsrc.eld in gzipped form is > something that should be explored. > > My .newsrc.eld on this netbook is 780kB, which isn't big by today's > network usage standards. Using "gzip --best" brings it down to 140kB, > which is... nothing, really. It's not much, but it would be nice to keep the network traffic down as much as possible, in these 3G days... -- (domestic pets only, the antidote for overdose, milk.) bloggy blog http://lars.ingebrigtsen.no/ ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 5:03 I can haz cloud idea Lars Ingebrigtsen ` (3 preceding siblings ...) 2012-02-16 13:07 ` Ted Zlatanov @ 2012-03-03 3:20 ` TSUCHIYA Masatoshi 2012-03-10 1:06 ` Lars Magne Ingebrigtsen 2014-10-15 23:31 ` TSUCHIYA Masatoshi 5 siblings, 1 reply; 40+ messages in thread From: TSUCHIYA Masatoshi @ 2012-03-03 3:20 UTC (permalink / raw) To: ding >> On Thu, 16 Feb 2012 06:03:47 +0100 >> larsi@gnus.org (Lars Ingebrigtsen) said as follows: >The main problems with going all cloudy is that you have to have yet >another server out there somewhere. Whether you use Dropbox/sshfs/a yet >to be determined Gnus-specific server, that's not very sexy. >But how about if we just cheat and put all the information on the IMAP >server? It sounds good. First of all, I want putting my draft box and an addressbook on the IMAP server. I hope that your Gnus cloud implementation may provide generic API to share files between machines. If such API will be available, I will be able to share the bookmark file and the list of visited URLs on emacs-w3m between machines. -- TSUCHIYA Masatoshi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-03-03 3:20 ` TSUCHIYA Masatoshi @ 2012-03-10 1:06 ` Lars Magne Ingebrigtsen 0 siblings, 0 replies; 40+ messages in thread From: Lars Magne Ingebrigtsen @ 2012-03-10 1:06 UTC (permalink / raw) To: ding TSUCHIYA Masatoshi <tsuchiya@namazu.org> writes: > First of all, I want putting my draft box and an addressbook on the IMAP > server. Yes, and Gnus should also start to defaulting to putting the archive group on the IMAP server (if the user is using nnimap at all). > I hope that your Gnus cloud implementation may provide generic API to > share files between machines. If such API will be available, I will be > able to share the bookmark file and the list of visited URLs on > emacs-w3m between machines. That was more or less my original idea, but, er, implementing Dropbox via IMAP had more conceptual sticky points that I liked. So I'm not currently thinking about doing a generic API. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog http://lars.ingebrigtsen.no/ ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2012-02-16 5:03 I can haz cloud idea Lars Ingebrigtsen ` (4 preceding siblings ...) 2012-03-03 3:20 ` TSUCHIYA Masatoshi @ 2014-10-15 23:31 ` TSUCHIYA Masatoshi 2014-10-16 8:34 ` Steinar Bang 2014-10-16 12:38 ` Ted Zlatanov 5 siblings, 2 replies; 40+ messages in thread From: TSUCHIYA Masatoshi @ 2014-10-15 23:31 UTC (permalink / raw) To: ding >> On Thu, 16 Feb 2012 06:03:47 +0100 >> larsi@gnus.org (Lars Ingebrigtsen) said as follows: >The main problems with going all cloudy is that you have to have yet >another server out there somewhere. Whether you use Dropbox/sshfs/a >yet to be determined Gnus-specific server, that's not very sexy. >But how about if we just cheat and put all the information on the IMAP >server? I use the following code to put my sent box on IMAP server. (let ((imap-method (catch 'found-imap-method (let ((methods (cons gnus-select-method gnus-secondary-select-methods))) (while methods (when (eq (caar methods) 'nnimap) (throw 'found-imap-method (car methods))) (setq methods (cdr methods))))))) (when imap-method (setq gnus-message-archive-method imap-method gnus-message-archive-group "Sent"))) I can share the same sent box between Gnus and other MUAs using this I also tried a similar trick to put my draft box on IMAP server, but failed. -- TSUCHIYA Masatoshi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2014-10-15 23:31 ` TSUCHIYA Masatoshi @ 2014-10-16 8:34 ` Steinar Bang 2014-10-16 12:38 ` Ted Zlatanov 1 sibling, 0 replies; 40+ messages in thread From: Steinar Bang @ 2014-10-16 8:34 UTC (permalink / raw) To: ding >>>>> TSUCHIYA Masatoshi <tsuchiya@namazu.org>: > I also tried a similar trick to put my draft box on IMAP server, but > failed. This could be because IMAP isn't rewritable...? Ie. on IMAP you would need to delete the old version of the message and write a new message with a new ID. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: I can haz cloud idea 2014-10-15 23:31 ` TSUCHIYA Masatoshi 2014-10-16 8:34 ` Steinar Bang @ 2014-10-16 12:38 ` Ted Zlatanov 1 sibling, 0 replies; 40+ messages in thread From: Ted Zlatanov @ 2014-10-16 12:38 UTC (permalink / raw) To: ding On Thu, 16 Oct 2014 08:31:54 +0900 TSUCHIYA Masatoshi <tsuchiya@namazu.org> wrote: >>> On Thu, 16 Feb 2012 06:03:47 +0100 >>> larsi@gnus.org (Lars Ingebrigtsen) said as follows: >> But how about if we just cheat and put all the information on the IMAP >> server? TM> I use the following code to put my sent box on IMAP server. Lars was talking about more than mail, we need all the metadata. Basically your newsrc.eld. Ted ^ permalink raw reply [flat|nested] 40+ messages in thread
end of thread, other threads:[~2014-10-16 12:38 UTC | newest] Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-02-16 5:03 I can haz cloud idea Lars Ingebrigtsen 2012-02-16 6:49 ` Lars Ingebrigtsen 2012-02-16 8:19 ` Antoine Levitt 2012-02-16 9:27 ` Steinar Bang 2012-02-16 9:35 ` Lars Ingebrigtsen 2012-02-16 7:00 ` Vegard Vesterheim 2012-02-17 17:49 ` Richard Riley 2012-02-19 15:21 ` Lars Ingebrigtsen 2012-02-19 18:04 ` Andy Moreton 2012-02-19 22:15 ` Dan Christensen 2012-02-20 7:41 ` Lars Ingebrigtsen 2012-02-20 19:20 ` Dan Christensen 2012-03-10 1:18 ` Lars Magne Ingebrigtsen 2012-02-20 7:37 ` Lars Ingebrigtsen 2012-02-20 7:51 ` Lars Ingebrigtsen 2012-02-19 19:03 ` Adam Sjøgren 2012-02-21 21:23 ` Ted Zlatanov 2012-03-10 1:12 ` Lars Magne Ingebrigtsen 2012-03-10 12:43 ` Reiner Steib 2012-03-10 13:06 ` Ted Zlatanov 2012-07-18 14:31 ` Ted Zlatanov 2012-07-18 20:52 ` Steinar Bang 2012-02-16 8:25 ` David Engster 2012-02-16 9:12 ` Lars Ingebrigtsen 2012-02-16 9:23 ` David Engster 2012-02-16 9:29 ` Steinar Bang 2012-02-16 9:25 ` Steinar Bang 2012-02-16 11:55 ` Greg Troxel 2012-02-16 12:23 ` David Engster 2012-02-16 12:54 ` Ted Zlatanov 2012-02-16 12:51 ` Ted Zlatanov 2012-02-16 13:07 ` Ted Zlatanov 2012-02-20 7:48 ` Lars Ingebrigtsen 2012-02-25 9:46 ` Steinar Bang 2012-03-10 1:07 ` Lars Magne Ingebrigtsen 2012-03-03 3:20 ` TSUCHIYA Masatoshi 2012-03-10 1:06 ` Lars Magne Ingebrigtsen 2014-10-15 23:31 ` TSUCHIYA Masatoshi 2014-10-16 8:34 ` Steinar Bang 2014-10-16 12:38 ` Ted Zlatanov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).