check mtime of newsrc.eld before saving it

check mtime of newsrc.eld before saving it

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

I sometimes update my newsrc.eld in a directory shared between multiple
machines.  I wanted a way to stop Gnus from overwriting the updated
file.

I came up with the following untested patch, which adds a customizable
option to do that check and saves the last modification time. But I
didn't finish testing or commit it because I had two questions:

1) is there a more elegant way to do this with file reverting?  It seems
like Emacs can usually tell if the file needs to be reverted.

2) should I make and use a new newsrc pre-save hook?

Let me know what you think...

Thanks
Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: save-newsrc-check.patch --]
[-- Type: text/x-patch, Size: 3071 bytes --]

diff --git a/lisp/gnus-start.el b/lisp/gnus-start.el
index 766e7c2..53e88a9 100644
--- a/lisp/gnus-start.el
+++ b/lisp/gnus-start.el
@@ -442,6 +442,14 @@ See also `gnus-before-startup-hook'."
   :group 'gnus-newsrc
   :type 'hook)
 
+(defcustom gnus-save-newsrc-file-check-timestamp nil
+  "Check the modification time of the newsrc.eld file before saving it.
+When the newsrc.eld file is updated by multiple machines,
+checking the file's modification time is a good way to avoid
+overwriting updated data."
+  :group 'gnus-newsrc
+  :type 'boolean)
+
 (defcustom gnus-save-newsrc-hook nil
   "A hook called before saving any of the newsrc files."
   :group 'gnus-newsrc
@@ -2783,6 +2791,7 @@ If FORCE is non-nil, the .newsrc file is read."
       'msdos-long-file-names
       (lambda () t))))
 
+(defvar gnus-save-newsrc-file-last-timestamp nil)
 (defun gnus-save-newsrc-file (&optional force)
   "Save .newsrc file."
   ;; Note: We cannot save .newsrc file if all newsgroups are removed
@@ -2821,12 +2830,24 @@ If FORCE is non-nil, the .newsrc file is read."
 	  (erase-buffer)
           (gnus-message 5 "Saving %s.eld..." gnus-current-startup-file)
 
+          ;; check timestamp of `gnus-current-startup-file'.eld against
+          ;; `gnus-save-newsrc-file-last-timestamp'
+          (when gnus-save-newsrc-file-check-timestamp
+            (let* ((checkfile (concat gnus-current-startup-file ".eld"))
+                   (mtime (nth 5 (file-attributes checkfile))))
+              (when (and gnus-save-newsrc-file-last-timestamp
+                         (time-less-p gnus-save-newsrc-file-last-timestamp
+                                      mtime))
+                (error "Couldn't save %s because it was updated" checkfile))))
+
           (if gnus-save-startup-file-via-temp-buffer
               (let ((coding-system-for-write gnus-ding-file-coding-system)
                     (standard-output (current-buffer)))
                 (gnus-gnus-to-quick-newsrc-format)
                 (gnus-run-hooks 'gnus-save-quick-newsrc-hook)
-                (save-buffer))
+                (save-buffer)
+                (setq gnus-save-newsrc-file-last-timestamp
+                            (nth 5 (file-attributes buffer-file-name))))
             (let ((coding-system-for-write gnus-ding-file-coding-system)
                   (version-control gnus-backup-startup-file)
                   (startup-file (concat gnus-current-startup-file ".eld"))
@@ -2861,7 +2882,9 @@ If FORCE is non-nil, the .newsrc file is read."
 
                       ;; Replace the existing startup file with the temp file.
                       (rename-file working-file startup-file t)
-                      (gnus-set-file-modes startup-file setmodes)))
+                      (gnus-set-file-modes startup-file setmodes)
+                      (setq gnus-save-newsrc-file-last-timestamp
+                            (nth 5 (file-attributes startup-file)))))
                 (condition-case nil
                     (delete-file working-file)
                   (file-error nil)))))

Re: check mtime of newsrc.eld before saving it

[Eric Abrahamsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I sometimes update my newsrc.eld in a directory shared between multiple
> machines.  I wanted a way to stop Gnus from overwriting the updated
> file.
>
> I came up with the following untested patch, which adds a customizable
> option to do that check and saves the last modification time. But I
> didn't finish testing or commit it because I had two questions:
>
> 1) is there a more elegant way to do this with file reverting?  It seems
> like Emacs can usually tell if the file needs to be reverted.
>
> 2) should I make and use a new newsrc pre-save hook?
>
> Let me know what you think...

I don't know too much about this, but got curious and went looking. So
far as I can tell, Emacs' clever file reversion/overwrite stuff is tied
to files that are visited by buffers (the buffer itself keeps track of
and compares last modification times), and not so much files that are
simply read and written. So I'm guessing your approach is probably the
right thing to do.

You probably knew all that already, but I learned something!

Eric

(elisp) Modification Time

Re: check mtime of newsrc.eld before saving it

[Ted Zlatanov]

On Tue, 07 Oct 2014 11:49:17 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 

EA> I don't know too much about this, but got curious and went looking. So
EA> far as I can tell, Emacs' clever file reversion/overwrite stuff is tied
EA> to files that are visited by buffers (the buffer itself keeps track of
EA> and compares last modification times), and not so much files that are
EA> simply read and written. So I'm guessing your approach is probably the
EA> right thing to do.

I could have used the buffer-specific stuff but there's actually another
version of the code that saves the newsrc, which uses file renames
instead of overwrites. So it got too complicated and I went back to my
original version. I added a y-or-n prompt and pushed to Git so it's
ready for use; just set `gnus-save-newsrc-file-check-timestamp' and
that's it.  If anyone can confirm it works all right for them, I will
add docs.

Ted

Re: check mtime of newsrc.eld before saving it

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I sometimes update my newsrc.eld in a directory shared between multiple
> machines.  I wanted a way to stop Gnus from overwriting the updated
> file.

[...]

> +(defcustom gnus-save-newsrc-file-check-timestamp nil
> +  "Check the modification time of the newsrc.eld file before saving it.

I like it, but I don't think we really need this variable.  Instead the
checking should always be on, and

[...]

> +                (error "Couldn't save %s because it was updated" checkfile))))

this should be a `y-or-n-p' instead, I think.  That would make Gnus'
behaviour here be pretty consistent with how Emacs otherwise behaves
when writing data to files.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: check mtime of newsrc.eld before saving it

[Ted Zlatanov]

On Tue, 27 Jan 2015 12:40:40 +1100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I sometimes update my newsrc.eld in a directory shared between multiple
>> machines.  I wanted a way to stop Gnus from overwriting the updated
>> file.

>> +(defcustom gnus-save-newsrc-file-check-timestamp nil
>> +  "Check the modification time of the newsrc.eld file before saving it.

LI> I like it, but I don't think we really need this variable.  Instead the
LI> checking should always be on

OK, I can make that change.  But then I need...

>> +                (error "Couldn't save %s because it was updated" checkfile))))

LI> this should be a `y-or-n-p' instead, I think.  That would make Gnus'
LI> behaviour here be pretty consistent with how Emacs otherwise behaves
LI> when writing data to files.

...a way to merge the on-disk newsrc.eld with the current one, because
currently aborting is the only way to avoid overwriting either side's
marks.

Something that walks all the groups and merges each entry would be
ideal; if the time of last group exit was recorded somewhere in the
marks, then this would be simply a timestamp compare.  Otherwise we may
have to prompt the user for any groups with different marks.  Could the
newsrc.eld format be modified to record this?

This is kind of related to the gnus-sync thing.

Ted

Re: check mtime of newsrc.eld before saving it

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> This is kind of related to the gnus-sync thing.

Yes.  Any chance of progress on that...?

Re: check mtime of newsrc.eld before saving it

[Ted Zlatanov]

On Wed, 04 Feb 2015 22:05:11 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> This is kind of related to the gnus-sync thing.

SB> Yes.  Any chance of progress on that...?

Yes. I need the IMAP backing in gnus-cloud.el that Lars was working on.

Ted

Re: check mtime of newsrc.eld before saving it

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LI> this should be a `y-or-n-p' instead, I think.  That would make Gnus'
> LI> behaviour here be pretty consistent with how Emacs otherwise behaves
> LI> when writing data to files.
>
> ...a way to merge the on-disk newsrc.eld with the current one, because
> currently aborting is the only way to avoid overwriting either side's
> marks.

Well, if you answer "yes" to "file change; write anyway?", Emacs will
overwrite changes made elsewhere to files normally.  So this would make
Gnus be pretty consistent with how Emacs works elsewhere, wouldn't it?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: check mtime of newsrc.eld before saving it

[Ted Zlatanov]

On Thu, 05 Feb 2015 14:13:09 +1100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
LI> this should be a `y-or-n-p' instead, I think.  That would make Gnus'
LI> behaviour here be pretty consistent with how Emacs otherwise behaves
LI> when writing data to files.
>> 
>> ...a way to merge the on-disk newsrc.eld with the current one, because
>> currently aborting is the only way to avoid overwriting either side's
>> marks.

LI> Well, if you answer "yes" to "file change; write anyway?", Emacs will
LI> overwrite changes made elsewhere to files normally.  So this would make
LI> Gnus be pretty consistent with how Emacs works elsewhere, wouldn't it?

Yes, but it's not what users want.  For me, at least (I count for two :)

The correct behavior would be to merge the newer newsrc as if it was a
dribble file.  I think

Ted

Re: check mtime of newsrc.eld before saving it

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> The correct behavior would be to merge the newer newsrc as if it was a
> dribble file.  I think

Yes, that would also be a good thing, but probably a separate issue
(that perhaps the cloud stuff may fix)...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: check mtime of newsrc.eld before saving it

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>> The correct behavior would be to merge the newer newsrc as if it was a
>> dribble file.  I think

> Yes, that would also be a good thing, but probably a separate issue
> (that perhaps the cloud stuff may fix)...

The cloud stuff, we wants it!  Yes we doesss!

Re: check mtime of newsrc.eld before saving it

[Ted Zlatanov]

On Tue, 27 Jan 2015 12:40:40 +1100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

>> +                (error "Couldn't save %s because it was updated" checkfile))))

LI> this should be a `y-or-n-p' instead, I think.  That would make Gnus'
LI> behaviour here be pretty consistent with how Emacs otherwise behaves
LI> when writing data to files.

I looked and there is a `y-or-n-p' just above.  The newsrc save flow is:

* if the timestamp is OK, proceed
* if the timestamp is newer, `y-or-n-p' to save
  * if yes, proceed (overwriting)
  * if no, error out

How should it work instead?

Ted

Re: check mtime of newsrc.eld before saving it

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I looked and there is a `y-or-n-p' just above.  The newsrc save flow is:
>
> * if the timestamp is OK, proceed
> * if the timestamp is newer, `y-or-n-p' to save
>   * if yes, proceed (overwriting)
>   * if no, error out

Hm...  I think that's the right work flow, indeed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Any cloudy news...? (Was: check mtime of newsrc.eld before saving it)

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> On Wed, 04 Feb 2015 22:05:11 +0100 Steinar Bang <sb@dod.no> wrote: 
>>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>>> This is kind of related to the gnus-sync thing.

SB> Yes.  Any chance of progress on that...?

> Yes. I need the IMAP backing in gnus-cloud.el that Lars was working on.

Any news about the cloudy stuff?

I upgraded to debian 8.0 "jessie" recently, which bumped couchdb from
1.2.0 to 1.4.0 and now gnus-sync.el doesn't work and I don't have sync
between my gnusen.

I could spend time debugging why gnus-sync.el doesn't work with couchdb
1.4.0, but if gnus-cloud is just around the corner, that would be a
waste of time and effort...? :-)

Trying to get gnus-sync working with plink (Was: Any cloudy news...?)

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> I upgraded to debian 8.0 "jessie" recently, which bumped couchdb from
> 1.2.0 to 1.4.0 and now gnus-sync.el doesn't work and I don't have sync
> between my gnusen.

> I could spend time debugging why gnus-sync.el doesn't work with
> couchdb 1.4.0, but if gnus-cloud is just around the corner, that would
> be a waste of time and effort...? :-)

I've given up on the couchdb route, all that happens when trying to
initialize couchdb 1.4.0 is that I'm getting strange error messages,
like e.g. a 404 on /sb/_design/lesync (my base is named "sb" and not
"tzz"), and /var/couchdb/couchdb.log fils up with JSON dumps with error
messages I don't understand.

So I'm trying to go the SSH file sync route, and I'm trying use plink as
the tramp method.

I have done this in .emacs:
(when windows-emacs
  (require 'tramp)
  (setenv "PATH" (concat "C:\\ProgramFiles\\PuTTY;" (getenv "PATH")))
  (setq tramp-default-method "plink"))

and I have done this in .gnus.el:
(setq gnus-sync-backend '("/sb@sync.mydomain.no:sync/gnus")
      gnus-sync-global-vars '(gnus-newsrc-last-checked-date)
      gnus-sync-newsrc-groups '("nntp" "nnrss")
      gnus-sync-newsrc-offsets '(2 3))

I have done
 C-x d /sb@sync.mydomain.no:sync RET
and successfully opened the remote directory using plink.

When I do
 C-u M-x gnus-sync-save RET
with these settings, nothing happens.

Any idea?  Do I have to make the sync file path explicitly use "plink:"?
Or do I need to use an ssh implementation with an executable called
ssh.exe (it doesn't look like there is anything in gnus-sync that cares,
as long as the file can be reached...?)

Re: Trying to get gnus-sync working with plink

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> So I'm trying to go the SSH file sync route, and I'm trying use plink as
> the tramp method.

> I have done this in .emacs:
> (when windows-emacs
>   (require 'tramp)
>   (setenv "PATH" (concat "C:\\ProgramFiles\\PuTTY;" (getenv "PATH")))
>   (setq tramp-default-method "plink"))

> and I have done this in .gnus.el:
> (setq gnus-sync-backend '("/sb@sync.mydomain.no:sync/gnus")
>       gnus-sync-global-vars '(gnus-newsrc-last-checked-date)
>       gnus-sync-newsrc-groups '("nntp" "nnrss")
>       gnus-sync-newsrc-offsets '(2 3))

The problem was that I had made gnus-sync-backend a list containing a
string, instead of just a string.  If I did it like this, it worked over
a plink tramp connection:
 (setq gnus-sync-backend '("/sb@sync.mydomain.no:sync/gnus")
       gnus-sync-global-vars '(gnus-newsrc-last-checked-date)
       gnus-sync-newsrc-groups '("nntp" "nnrss")
       gnus-sync-newsrc-offsets '(2 3))

Very slow compared to the lesync version, though.

Re: Trying to get gnus-sync working with plink

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>> So I'm trying to go the SSH file sync route, and I'm trying use plink as
>> the tramp method.

>> I have done this in .emacs:
>> (when windows-emacs
>> (require 'tramp)
>> (setenv "PATH" (concat "C:\\ProgramFiles\\PuTTY;" (getenv "PATH")))
>> (setq tramp-default-method "plink"))

>> and I have done this in .gnus.el:
>> (setq gnus-sync-backend '("/sb@sync.mydomain.no:sync/gnus")
>> gnus-sync-global-vars '(gnus-newsrc-last-checked-date)
>> gnus-sync-newsrc-groups '("nntp" "nnrss")
>> gnus-sync-newsrc-offsets '(2 3))

> The problem was that I had made gnus-sync-backend a list containing a
> string, instead of just a string.  If I did it like this, it worked over
> a plink tramp connection:

Forgot to correct the lisp code, this is what works:
  (setq gnus-sync-backend "/sb@sync.mydomain.no:sync/gnus"
        gnus-sync-global-vars '(gnus-newsrc-last-checked-date)
        gnus-sync-newsrc-groups '("nntp" "nnrss")
        gnus-sync-newsrc-offsets '(2 3))

> Very slow compared to the lesync version, though.

Not really.  I just misinterpreted the feedback from the initial save
and thought it was waiting for a response when it was actually finished.

Re: Trying to get gnus-sync working with plink

[Sivaram Neelakantan]

On Mon, Jun 08 2015,Steinar Bang wrote:

>>>>>> Steinar Bang <sb@dod.no>:
>

[snipped 24 lines]

>
>> Very slow compared to the lesync version, though.
>
> Not really.  I just misinterpreted the feedback from the initial save
> and thought it was waiting for a response when it was actually finished.
>
>
>

Could you post the full setup and how you got it working, please?

 sivaram
 -- 

Re: Trying to get gnus-sync working with plink

[Steinar Bang]

>>>>> Sivaram Neelakantan <nsivaram.net@gmail.com>:

> Could you post the full setup and how you got it working, please?

First:
 - You need a server you can SSH into
 - Make sure PuTTY works against this server using public/private key
   authentication (I do it without using a passphrase, no idea what
   would be needed to make it work with a passphrase)

Setting up PuTTY in this way is left as an exercise for the reader.

However using the value for gnus-sync-backend below, the "Host Name (or
IP address)" field in "Session" should be: username@sync.mydomain.no

The session name in "Saved Sessions" should be: sync.mydomain.no

The session is then saved.

Secondly the setup og emacs is as follows:
 1. Adding plink.exe to emacs's PATH and making plink be the default
    remote file method, put this in ~/.emacs:
      (require 'tramp)
      (setenv "PATH" (concat "C:\\ProgramFiles\\PuTTY;" (getenv "PATH")))
      (setq tramp-default-method "plink")
 2. Put the following in ~/.gnus.el:
      (require 'gnus-sync)
      (setq gnus-sync-backend "/username@sync.mydomain.no:sync/gnus"
            gnus-sync-global-vars '(gnus-newsrc-last-checked-date)
            gnus-sync-newsrc-groups '("nntp" "nnrss")
            gnus-sync-newsrc-offsets '(2 3))
 3. Make sure "gzip" is in emacs' PATH (I got it from GnuWin32), which I
    do the following way:
      (setenv "PATH" (concat "c:\\ProgramFiles\\ezwinports\\bin;C:\\ProgramFiles\\GnuWin32\\bin;" (getenv "PATH"))

To initially save all NNTP groups, do: C-u M-x gnus-sync-save RET
Later
 - save with: M-x gnus-sync-save RET
 - load with: M-x gnus-sync-read RET
   - After the gnus-sync-read command has completed, to 'g' in the
     *Group* buffer, and the read numbers will be adjusted

gnus-sync work (was: Trying to get gnus-sync working with plink)

[Ted Zlatanov]

On Sun, 07 Jun 2015 23:28:16 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> I've given up on the couchdb route, all that happens when trying to
SB> initialize couchdb 1.4.0 is that I'm getting strange error messages,
SB> like e.g. a 404 on /sb/_design/lesync (my base is named "sb" and not
SB> "tzz"), and /var/couchdb/couchdb.log fils up with JSON dumps with error
SB> messages I don't understand.

Yeah... I haven't used this in years and CouchDB has not been a fun
environment. It was a mistake for me to rely on it.

Lars had agreed to use IMAP as the sync backend and started work on
that, but I don't know the current status (I ask this question once a
year, sorry). The ultimate goal is to host the newsrc.eld entirely
inside IMAP. My work with Tramp to support file access over IMAP (now
removed because it was not used by anyone) could have been part of that,
but clearly an atomic file would not work, we need to break out the
records for each group so they can be updated independently.

Using the Gnus slave code seems clever but would be a mistake, I think
(because the assumption with Gnus slaves is that a master will
eventually come along and gather the dribble files). Another way of
looking at it is that Gnus needs to be in perpetual slave mode, there
should be no master.

In a modern environment, I think there are many choices for distributed
databases but I think IMAP remains the best way to synchronize data
across multiple news readers even though it's a terrible data protocol.
Partly that's because it already synchronizes mail through article
marks. But I'm not sure, honestly.

Opinions welcome.

Ted

Re: gnus-sync work

[Dave Abrahams]

on Mon Mar 07 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Sun, 07 Jun 2015 23:28:16 +0200 Steinar Bang <sb@dod.no> wrote: 
>
> SB> I've given up on the couchdb route, all that happens when trying to
> SB> initialize couchdb 1.4.0 is that I'm getting strange error messages,
> SB> like e.g. a 404 on /sb/_design/lesync (my base is named "sb" and not
> SB> "tzz"), and /var/couchdb/couchdb.log fils up with JSON dumps with error
> SB> messages I don't understand.
>
> Yeah... I haven't used this in years and CouchDB has not been a fun
> environment. It was a mistake for me to rely on it.
>
> Lars had agreed to use IMAP as the sync backend and started work on
> that, but I don't know the current status (I ask this question once a
> year, sorry). The ultimate goal is to host the newsrc.eld entirely
> inside IMAP. My work with Tramp to support file access over IMAP (now
> removed because it was not used by anyone) could have been part of that,
> but clearly an atomic file would not work, we need to break out the
> records for each group so they can be updated independently.
>
> Using the Gnus slave code seems clever but would be a mistake, I think
> (because the assumption with Gnus slaves is that a master will
> eventually come along and gather the dribble files). Another way of
> looking at it is that Gnus needs to be in perpetual slave mode, there
> should be no master.
>
> In a modern environment, I think there are many choices for distributed
> databases but I think IMAP remains the best way to synchronize data
> across multiple news readers even though it's a terrible data protocol.
> Partly that's because it already synchronizes mail through article
> marks. But I'm not sure, honestly.

I wonder how ridiculous it would be to, for every newsgroup, create imap
folders with empty shadow articles for each news article?  Then you
could access NNTP and IMAP in parallel and just use IMAP for the article
marks.

Insane?

-- 
-Dave

Re: gnus-sync work

[Steinar Bang]

>>>>> Dave Abrahams <dave@boostpro.com>:

> I wonder how ridiculous it would be to, for every newsgroup, create
> imap folders with empty shadow articles for each news article?  Then
> you could access NNTP and IMAP in parallel and just use IMAP for the
> article marks.

The problem will be synching the article numbers in an efficient manner
(ie. based on the NOV data), I think...?

Or...? Perhaps not...?  Is the message-id a part of the NOV data?  That
would be identical in both folders, I think...?

Re: gnus-sync work

[Ted Zlatanov]

On Fri, 20 May 2016 19:43:57 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Dave Abrahams <dave@boostpro.com>:
>> I wonder how ridiculous it would be to, for every newsgroup, create
>> imap folders with empty shadow articles for each news article?  Then
>> you could access NNTP and IMAP in parallel and just use IMAP for the
>> article marks.

I think that's not a bad idea, but it creates a lot of opportunities for
things to get out of sync and would be very slow (using these shadow
articles requires spooling them, waiting, then setting their marks).
Entering a large group could take hours.

Lars' idea, from when we talked about it, was to create articles with a
data description of what has changed since the last full sync, so you
can read the articles to recreate the marks.

SB> The problem will be synching the article numbers in an efficient manner
SB> (ie. based on the NOV data), I think...?

SB> Or...? Perhaps not...?  Is the message-id a part of the NOV data?  That
SB> would be identical in both folders, I think...?

How about this protocol:

* designate an IMAP folder as the sync home
* message subject is the group name
* message body is just a dump of the Lisp data for the marks
* messages marked Important (ticked) are full syncs of the marks for that group
* messages not marked Important are differential syncs (protocol to be determined, let's stick to full sync only)

This would be fast and you can even go back to older marks if you catch
up by accident or something. The marks would be stored in a small
consistent data store, atomically updated (most recent wins).

Ted

Re: gnus-sync work

[Dave Abrahams]

on Tue May 24 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Fri, 20 May 2016 19:43:57 +0200 Steinar Bang <sb@dod.no> wrote: 
>
>>>>>>> Dave Abrahams <dave@boostpro.com>:
>>> I wonder how ridiculous it would be to, for every newsgroup, create
>>> imap folders with empty shadow articles for each news article?  Then
>>> you could access NNTP and IMAP in parallel and just use IMAP for the
>>> article marks.
>
> I think that's not a bad idea, but it creates a lot of opportunities for
> things to get out of sync and would be very slow (using these shadow
> articles requires spooling them, waiting, then setting their marks).
> Entering a large group could take hours.
>
> Lars' idea, from when we talked about it, was to create articles with a
> data description of what has changed since the last full sync, so you
> can read the articles to recreate the marks.
>
> SB> The problem will be synching the article numbers in an efficient manner
> SB> (ie. based on the NOV data), I think...?
>
> SB> Or...? Perhaps not...?  Is the message-id a part of the NOV data?  That
> SB> would be identical in both folders, I think...?
>
> How about this protocol:
>
> * designate an IMAP folder as the sync home
> * message subject is the group name
> * message body is just a dump of the Lisp data for the marks
> * messages marked Important (ticked) are full syncs of the marks for that group
> * messages not marked Important are differential syncs (protocol to be determined, let's stick to full sync only)
>
> This would be fast and you can even go back to older marks if you catch
> up by accident or something. The marks would be stored in a small
> consistent data store, atomically updated (most recent wins).

SGTM!  Want! :-)

-- 
Dave

Re: gnus-sync work

[Steinar Bang]

>>>>> Dave Abrahams <dave@boostpro.com>:

> on Tue May 24 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

>> How about this protocol:
>> 
>> * designate an IMAP folder as the sync home
>> * message subject is the group name
>> * message body is just a dump of the Lisp data for the marks
>> * messages marked Important (ticked) are full syncs of the marks for that group
>> * messages not marked Important are differential syncs (protocol to be determined, let's stick to full sync only)
>> 
>> This would be fast and you can even go back to older marks if you catch
>> up by accident or something. The marks would be stored in a small
>> consistent data store, atomically updated (most recent wins).

> SGTM!  Want! :-)

AOL! (ie. "Me too!")

It would actually make me git clone emacs, instead of using the latest
released version.

How stable is the git emacs?  Perhaps I would end up using it only for Gnus?

Re: gnus-sync work

[Dave Abrahams]

on Tue May 31 2016, Dave Abrahams <dave-AT-boostpro.com> wrote:

> on Tue May 24 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:
>
>> On Fri, 20 May 2016 19:43:57 +0200 Steinar Bang <sb@dod.no> wrote: 
>>
>>>>>>>> Dave Abrahams <dave@boostpro.com>:
>>>> I wonder how ridiculous it would be to, for every newsgroup, create
>>>> imap folders with empty shadow articles for each news article?  Then
>>>> you could access NNTP and IMAP in parallel and just use IMAP for the
>>>> article marks.
>>
>> I think that's not a bad idea, but it creates a lot of opportunities for
>> things to get out of sync and would be very slow (using these shadow
>> articles requires spooling them, waiting, then setting their marks).
>> Entering a large group could take hours.
>>
>> Lars' idea, from when we talked about it, was to create articles with a
>> data description of what has changed since the last full sync, so you
>> can read the articles to recreate the marks.
>>
>> SB> The problem will be synching the article numbers in an efficient manner
>> SB> (ie. based on the NOV data), I think...?
>>
>> SB> Or...? Perhaps not...?  Is the message-id a part of the NOV data?  That
>> SB> would be identical in both folders, I think...?
>>
>> How about this protocol:
>>
>> * designate an IMAP folder as the sync home
>> * message subject is the group name
>> * message body is just a dump of the Lisp data for the marks
>> * messages marked Important (ticked) are full syncs of the marks for that group
>> * messages not marked Important are differential syncs (protocol to be determined, let's stick to full sync only)
>>
>> This would be fast and you can even go back to older marks if you catch
>> up by accident or something. The marks would be stored in a small
>> consistent data store, atomically updated (most recent wins).
>
> SGTM!  Want! :-)

I don't mean to be pushy, but realistically speaking, is there any
chance this will happen in the next few weeks?  I am seriously thinking
of moving most of my NNTP participation to GMail/IMAP just to avoid the
sync issue.

-- 
-Dave

Re: gnus-sync work

[Ted Zlatanov]

On Sun, 05 Jun 2016 13:27:35 -0700 Dave Abrahams <dave@boostpro.com> wrote: 

DA> I don't mean to be pushy, but realistically speaking, is there any
DA> chance this will happen in the next few weeks?  I am seriously thinking
DA> of moving most of my NNTP participation to GMail/IMAP just to avoid the
DA> sync issue.

I want to do it, but really want to know what Lars thinks because I know
he had started working on something similar already.

If not, this weekend I'll bang it out as a new gnus-sync.el backend (and
obsolete the old CouchDB one).

The other issue is that the Emacs release is imminent so we're in a
feature freeze. What branch do I use and can I push changes at all? I
could make a feature branch if Dave, Steinar, and others are willing to
test that.

On Wed, 01 Jun 2016 07:38:16 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> It would actually make me git clone emacs, instead of using the latest
SB> released version.

SB> How stable is the git emacs?  Perhaps I would end up using it only for Gnus?

It's very stable for me on Linux. But you can probably adjust the load
path so gnus-sync.el comes from Git, but the rest doesn't. I wouldn't
bother, personally.

Ted

Re: gnus-sync work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

SB> How stable is the git emacs?  Perhaps I would end up using it only
SB> for Gnus?

> It's very stable for me on Linux. But you can probably adjust the load
> path so gnus-sync.el comes from Git, but the rest doesn't. I wouldn't
> bother, personally.

Yeah, but I would need it on two to three windoze machines, and one
debian machine.

The debian machine would probably be the least troublesome one.

We'll see, I guess.

gnus-cloud work (was: gnus-sync work)

[Ted Zlatanov]

On Tue, 07 Jun 2016 14:22:27 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Sun, 05 Jun 2016 13:27:35 -0700 Dave Abrahams <dave@boostpro.com> wrote: 
DA> I don't mean to be pushy, but realistically speaking, is there any
DA> chance this will happen in the next few weeks?  I am seriously thinking
DA> of moving most of my NNTP participation to GMail/IMAP just to avoid the
DA> sync issue.

TZ> I want to do it, but really want to know what Lars thinks because I know
TZ> he had started working on something similar already.

TZ> If not, this weekend I'll bang it out as a new gnus-sync.el backend (and
TZ> obsolete the old CouchDB one).

All right, after looking at gnus-cloud.el, I think I understand where
Lars was going. Unfortunately he hasn't touched that code in years so we
have to get going with what's there. In any case, it's better code than
gnus-sync.el, especially as far as internal glue is concerned, so
gnus-sync.el is going away.

gnus-cloud.el did:

* set up a mechanism where every server could be "cloud synced"
* have the concept of a "cloud host server+group"
* save sequences representing files or pure data to IMAP articles
* have ambitions to be Emacs-wide, so perhaps should move out of Gnus?

My changes now are in the Emacs Savannah repo in branch scratch/gnus-cloud:

* rename the cloud host group name to "Emacs-Cloud" so it works with GMail
* add `I' and `gnus-server-toggle-cloud-method-server' in the Server buffer to set an IMAP server as the cloud host and do an initial full upload
* show the cloud host distinctly visually
* remove gnus-sync.el
* a full upload works

To try it: switch to the scratch/gnus-cloud branch, go to the Server
buffer, and hit `I' or `M-x gnus-server-toggle-cloud-method-server'

Try this outside a server line: you'll get an error.
Try this on a non-IMAP server: you'll get an error.
Try this on an IMAP server: you should see `CLOUD-HOST' on it, and the
initial upload should run and complete successfully.

Still remaining:

* verify uploads
* download cloud data back
* when a server is selected as the CLOUD-HOST, maybe save that choice? but I'm not sure if the choice should be persisted in `gnus-cloud-method' or in the newsrc.eld.
* notice when the cloud data changes (maybe some IMAP magic can do it... but could just be done explicitly for now)
* use PGP instead of Base64+gzip (could be a choice... but I would
* really prefer to make it secure by default and make insecure storage an explicit user choice)
* get Steinar, Dave, and others to try this out...
* send salmon hunter drones after Lars to get him back on this :)

Ted

Re: gnus-cloud work

[Ted Zlatanov]

On Wed, 15 Jun 2016 14:55:18 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> * verify uploads
TZ> * download cloud data back
TZ> * use PGP instead of Base64+gzip (could be a choice... but I would
TZ> * really prefer to make it secure by default and make insecure storage an explicit user choice)

...done

TZ> * get Steinar, Dave, and others to try this out...
TZ> * send salmon hunter drones after Lars to get him back on this :)
TZ> * when a server is selected as the CLOUD-HOST, maybe save that choice? but I'm not sure if the choice should be persisted in `gnus-cloud-method' or in the newsrc.eld.
TZ> * notice when the cloud data changes (maybe some IMAP magic can do it... but could just be done explicitly for now)

...not done

I also changed some headers etc.

emacs.git, branch scratch/gnus-cloud

Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

TZ> * get Steinar, Dave, and others to try this out...

I'm committed to trying this out, if I can find a way that doesn't
involve me moving to git emacs.

(Currently I'm trying to move your changes to obsolete-gnus as patches,
but the path-doctored patches won't apply.  I think the problem is that
I tried to do this on a windows PC.  I will try on my debian computer.
(If someone can come up with the proper subtree merge invocations I
could try that instead of patches, but manually doctored patches seemed
simplest))

Re: gnus-cloud work

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> (Currently I'm trying to move your changes to obsolete-gnus as patches,
> but the path-doctored patches won't apply.  I think the problem is that
> I tried to do this on a windows PC.  I will try on my debian computer.
> (If someone can come up with the proper subtree merge invocations I
> could try that instead of patches, but manually doctored patches seemed
> simplest))

Didn't work on the debian computer either.

Here's what I did:
 1. Cloned emacs
 2. Switched to the scratch/gnus-cloud branch
     git checkout scratch/gnus-cloud
 3. Created a patch from the commit at the end of the branch
     git format-patch -1 743e6b6
 4. Edited the patch, replacing "lisp/gnus/" with "lisp/" to match the
    layout of the gnus repository
 5. Tried to apply the patch to gnus
     cd ../gnus
     git apply ../emacs/0001-Gnus-Emacs-Cloud-fixes-and-refactoring.patch

But the patch failed to apply:
 error: patch failed: lisp/gnus-srvr.el:306
 error: lisp/gnus-srvr.el: patch does not apply
 error: patch failed: lisp/gnus-sync.el:1
 error: lisp/gnus-sync.el: patch does not apply

Does anyone know to doctor the patch correctly?

Re: gnus-cloud work

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>  5. Tried to apply the patch to gnus
>      cd ../gnus
>      git apply ../emacs/0001-Gnus-Emacs-Cloud-fixes-and-refactoring.patch

> But the patch failed to apply:
>  error: patch failed: lisp/gnus-srvr.el:306
>  error: lisp/gnus-srvr.el: patch does not apply
>  error: patch failed: lisp/gnus-sync.el:1
>  error: lisp/gnus-sync.el: patch does not apply

I tried a different approach:
 cd ../gnus/lisp
 git apply -p2 ../../emacs/0001-Gnus-Emacs-Cloud-fixes-and-refactoring.patch

There was no output and no changes made to the files, and no extra
commit made.

If I tried stripping just one directory level I got error messages:
 sb@lorenzo:~/git/gnus/lisp$ git apply -p1 ../../emacs/0001-Gnus-Emacs-Cloud-fixes-and-refactoring.patch
 error: lisp/gnus/gnus-cloud.el: No such file or directory
 error: lisp/gnus/gnus-srvr.el: No such file or directory
 error: lisp/gnus/gnus-sync.el: No such file or directory

Re: gnus-cloud work

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Steinar Bang <sb@dod.no>:
>> 5. Tried to apply the patch to gnus
>>    cd ../gnus
>>    git apply ../emacs/0001-Gnus-Emacs-Cloud-fixes-and-refactoring.patch

>> But the patch failed to apply:
>> error: patch failed: lisp/gnus-srvr.el:306
>> error: lisp/gnus-srvr.el: patch does not apply
>> error: patch failed: lisp/gnus-sync.el:1
>> error: lisp/gnus-sync.el: patch does not apply

I found the reason the patch didn't apply. I did
 git apply -v ../emacs/0001-Gnus-Emacs-Cloud-fixes-and-refactoring.patch
on the file with the doctored patches, and that told me that the patches
to gnus-srvr.el didn't apply because the file has changed too much.

The gnus-srvr.el file has many changes postdating its move to the emacs
git repository.

Re: gnus-cloud work

[Andreas Schwab]

Steinar Bang <sb@dod.no> writes:

> I found the reason the patch didn't apply. I did
>  git apply -v ../emacs/0001-Gnus-Emacs-Cloud-fixes-and-refactoring.patch
> on the file with the doctored patches, and that told me that the patches
> to gnus-srvr.el didn't apply because the file has changed too much.

Perhaps git apply -3 can help.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: gnus-cloud work

[Steinar Bang]

>>>>> Andreas Schwab <schwab@linux-m68k.org>:

> Perhaps git apply -3 can help.

Nope.  Failed the same way, except for an extra error message:
 error: repository lacks the necessary blob to fall back on 3-way merge.

Re: gnus-cloud work

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Ted Zlatanov <tzz@lifelogs.com>:
TZ> * get Steinar, Dave, and others to try this out...

> I'm committed to trying this out, if I can find a way that doesn't
> involve me moving to git emacs.

Ok, now I've tried it out:
 1. Entered the *server buffer*
 2. Positioned the cursor in front of the nnimap server
 3. Typed I
 4. Got the following error message:
 Debugger entered--Lisp error: (error "Format specifier doesn't match argument type")
  format("Subject: (sequence: %d type: %s storage-method: %s)\n" "nil" :full epg)
  (insert (format "Subject: (sequence: %d type: %s storage-method: %s)\n" gnus-cloud-sequence (if full :full :partial) gnus-cloud-storage-method))
  (let ((elems (gnus-cloud-files-to-upload full)) (group (gnus-group-full-name gnus-cloud-group-name gnus-cloud-method))) (insert (format "Subject: (sequence: %d type: %s storage-method: %s)\n" gnus-cloud-sequence (if full :full :partial) gnus-cloud-storage-method)) (insert "From: nobody@gnus.cloud.invalid\n") (insert "\n") (insert (gnus-cloud-make-chunk elems)) (if (gnus-request-accept-article gnus-cloud-group-name gnus-cloud-method t t) (progn (setq gnus-cloud-sequence (1+ gnus-cloud-sequence)) (gnus-cloud-add-timestamps elems) (gnus-message 3 "Uploaded Emacs Cloud data successfully to %s" group)) (gnus-error 2 "Failed to upload Emacs Cloud data to %s" group)))
  (progn (let ((elems (gnus-cloud-files-to-upload full)) (group (gnus-group-full-name gnus-cloud-group-name gnus-cloud-method))) (insert (format "Subject: (sequence: %d type: %s storage-method: %s)\n" gnus-cloud-sequence (if full :full :partial) gnus-cloud-storage-method)) (insert "From: nobody@gnus.cloud.invalid\n") (insert "\n") (insert (gnus-cloud-make-chunk elems)) (if (gnus-request-accept-article gnus-cloud-group-name gnus-cloud-method t t) (progn (setq gnus-cloud-sequence (1+ gnus-cloud-sequence)) (gnus-cloud-add-timestamps elems) (gnus-message 3 "Uploaded Emacs Cloud data successfully to %s" group)) (gnus-error 2 "Failed to upload Emacs Cloud data to %s" group))))
  (unwind-protect (progn (let ((elems (gnus-cloud-files-to-upload full)) (group (gnus-group-full-name gnus-cloud-group-name gnus-cloud-method))) (insert (format "Subject: (sequence: %d type: %s storage-method: %s)\n" gnus-cloud-sequence (if full :full :partial) gnus-cloud-storage-method)) (insert "From: nobody@gnus.cloud.invalid\n") (insert "\n") (insert (gnus-cloud-make-chunk elems)) (if (gnus-request-accept-article gnus-cloud-group-name gnus-cloud-method t t) (progn (setq gnus-cloud-sequence (1+ gnus-cloud-sequence)) (gnus-cloud-add-timestamps elems) (gnus-message 3 "Uploaded Emacs Cloud data successfully to %s" group)) (gnus-error 2 "Failed to upload Emacs Cloud data to %s" group)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((elems (gnus-cloud-files-to-upload full)) (group (gnus-group-full-name gnus-cloud-group-name gnus-cloud-method))) (insert (format "Subject: (sequence: %d type: %s storage-method: %s)\n" gnus-cloud-sequence (if full :full :partial) gnus-cloud-storage-method)) (insert "From: nobody@gnus.cloud.invalid\n") (insert "\n") (insert (gnus-cloud-make-chunk elems)) (if (gnus-request-accept-article gnus-cloud-group-name gnus-cloud-method t t) (progn (setq gnus-cloud-sequence (1+ gnus-cloud-sequence)) (gnus-cloud-add-timestamps elems) (gnus-message 3 "Uploaded Emacs Cloud data successfully to %s" group)) (gnus-error 2 "Failed to upload Emacs Cloud data to %s" group)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
 )
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((elems (gnus-cloud-files-to-upload full)) (group (gnus-group-full-name gnus-cloud-group-name gnus-cloud-method))) (insert (format "Subject: (sequence: %d type: %s storage-method: %s)\n" gnus-cloud-sequence (if full :full :partial) gnus-cloud-storage-method)) (insert "From: nobody@gnus.cloud.invalid\n") (insert "\n") (insert (gnus-cloud-make-chunk elems)) (if (gnus-request-accept-article gnus-cloud-group-name gnus-cloud-method t t) (progn (setq gnus-cloud-sequence ...) (gnus-cloud-add-timestamps elems) (gnus-message 3 "Uploaded Emacs Cloud data successfully to %s" group)) (gnus-error 2 "Failed to upload Emacs Cloud data to %s" group)))) (and (buffer-name temp-buf
 fer) (kill-buffer temp-buffer)))))
  gnus-cloud-upload-data(t)
  gnus-server-toggle-cloud-method-server()
  call-interactively(gnus-server-toggle-cloud-method-server nil nil)
  command-execute(gnus-server-toggle-cloud-method-server)

The server is marked with GNUS-CLOUD, I don't know what else to look
for.

Note: I'm running gnus-obsolete with your changes, not emacs git gnus.

I gave up applying the patches, just copied the modified files over from
the emacs repo.

The git repository is here:
 https://github.com/steinarb/gnus-obsolete-gnus-cloud

Re: gnus-cloud work

[Ted Zlatanov]

On Mon, 20 Jun 2016 14:21:12 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
TZ> * get Steinar, Dave, and others to try this out...

SB> I'm committed to trying this out, if I can find a way that doesn't
SB> involve me moving to git emacs.

Do it! Do it!

Ted

Re: gnus-cloud work

[Ted Zlatanov]

On Tue, 21 Jun 2016 20:52:51 +0200 Steinar Bang <sb@dod.no> wrote: 

SB>  Debugger entered--Lisp error: (error "Format specifier doesn't match argument type")
SB>   format("Subject: (sequence: %d type: %s storage-method: %s)\n" "nil" :full epg)
SB>   (insert (format "Subject: (sequence: %d type: %s storage-method: %s)\n" gnus-cloud-sequence (if full :full :partial) gnus-cloud-storage-method))

Strange, I don't get that error. I just tested with a clean install.

Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> On Tue, 21 Jun 2016 20:52:51 +0200 Steinar Bang <sb@dod.no> wrote: 
SB> Debugger entered--Lisp error: (error "Format specifier doesn't match argument type")
SB> format("Subject: (sequence: %d type: %s storage-method: %s)\n" "nil" :full epg)
SB> (insert (format "Subject: (sequence: %d type: %s storage-method: %s)\n" gnus-cloud-sequence (if full :full :partial) gnus-cloud-storage-method))

> Strange, I don't get that error. I just tested with a clean install.

Yeah, but you're on git emacs, right?  

I'm in emacs 24.4, on my own gnus-obsolete-with-your-gnus-cloud-changes-applied:
 https://github.com/steinarb/gnus-obsolete-gnus-cloud
(well, your changes as well as all gnus-in-emacs-git changes made to
gnus-srvr.el)

Also, I did my test on Windows 7, while you may be on GNU/linux?

Re: gnus-cloud work

[Ted Zlatanov]

On Mon, 27 Jun 2016 17:07:20 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> On Tue, 21 Jun 2016 20:52:51 +0200 Steinar Bang <sb@dod.no> wrote: 
SB> Debugger entered--Lisp error: (error "Format specifier doesn't match argument type")
SB> format("Subject: (sequence: %d type: %s storage-method: %s)\n" "nil" :full epg)
SB> (insert (format "Subject: (sequence: %d type: %s storage-method: %s)\n" gnus-cloud-sequence (if full :full :partial) gnus-cloud-storage-method))

>> Strange, I don't get that error. I just tested with a clean install.

SB> Yeah, but you're on git emacs, right?  

SB> I'm in emacs 24.4, on my own gnus-obsolete-with-your-gnus-cloud-changes-applied:
SB>  https://github.com/steinarb/gnus-obsolete-gnus-cloud
SB> (well, your changes as well as all gnus-in-emacs-git changes made to
SB> gnus-srvr.el)

Right. You're going to have to use the Git Emacs checkout.

SB> Also, I did my test on Windows 7, while you may be on GNU/linux?

That shouldn't matter for the gnus-cloud work, but I don't know if the
EPA/EPG tools work well there. You may have to change
`gnus-cloud-storage-method' which defaults to 'epg if EPA/EPG is loaded:

gnus-cloud-storage-method is a variable defined in ‘gnus-cloud.el’.
Its value is ‘epg’

If you have gzip available, set it to Base64+gzip. Otherwise just Base64
should be enough.

Ted

Re: gnus-cloud work

[Eric Abrahamsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 15 Jun 2016 14:55:18 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 
>
> TZ> * verify uploads
> TZ> * download cloud data back
> TZ> * use PGP instead of Base64+gzip (could be a choice... but I would
> TZ> * really prefer to make it secure by default and make insecure storage an explicit user choice)
>
> ...done
>
> TZ> * get Steinar, Dave, and others to try this out...
> TZ> * send salmon hunter drones after Lars to get him back on this :)
> TZ> * when a server is selected as the CLOUD-HOST, maybe save that
> TZ> choice? but I'm not sure if the choice should be persisted in
> TZ> `gnus-cloud-method' or in the newsrc.eld.
> TZ> * notice when the cloud data changes (maybe some IMAP magic can do it... but could just be done explicitly for now)
>
> ...not done
>
> I also changed some headers etc.

Hey I think I'm confused about what this is meant to do. My original
understanding was that this was for syncing NNTP marks between machines,
using an IMAP server. Is that what this is for?

I have a laptop and a desktop. Both are running this branch, and on both
I've marked the same IMAP server as "cloud-sync".

But what happens now?

E

Re: gnus-cloud work

[Ted Zlatanov]

On Thu, 30 Jun 2016 22:42:22 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 

EA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Wed, 15 Jun 2016 14:55:18 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 
>> 
TZ> * verify uploads
TZ> * download cloud data back
TZ> * use PGP instead of Base64+gzip (could be a choice... but I would
TZ> * really prefer to make it secure by default and make insecure storage an explicit user choice)
>> 
>> ...done
>> 
TZ> * get Steinar, Dave, and others to try this out...
TZ> * send salmon hunter drones after Lars to get him back on this :)
TZ> * when a server is selected as the CLOUD-HOST, maybe save that
TZ> choice? but I'm not sure if the choice should be persisted in
TZ> `gnus-cloud-method' or in the newsrc.eld.
TZ> * notice when the cloud data changes (maybe some IMAP magic can do it... but could just be done explicitly for now)
>> 
>> ...not done
>> 
>> I also changed some headers etc.

EA> Hey I think I'm confused about what this is meant to do. My original
EA> understanding was that this was for syncing NNTP marks between machines,
EA> using an IMAP server. Is that what this is for?

Yes. Plus files, optionally.

EA> I have a laptop and a desktop. Both are running this branch, and on both
EA> I've marked the same IMAP server as "cloud-sync".

EA> But what happens now?

Upload your first cloud pack from one machine and see if that works:
`gnus-cloud-upload-data'

You should see an IMAP article containing the cloud pack in a new
Emacs-Cloud IMAP folder. If you used EPG encryption, it should be
appropriately encrypted and ASCII armored. Otherwise, it will be
Base64+gzip. Internally it's just Lisp data.

That's where I left things; downloading is not done but should at least
find the article and parse it with `gnus-cloud-download-data'.

Ted

Re: gnus-cloud work

[Eric Abrahamsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 30 Jun 2016 22:42:22 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 
>
> EA> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> On Wed, 15 Jun 2016 14:55:18 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 
>>> 
> TZ> * verify uploads
> TZ> * download cloud data back
> TZ> * use PGP instead of Base64+gzip (could be a choice... but I would
> TZ> * really prefer to make it secure by default and make insecure storage an explicit user choice)
>>> 
>>> ...done
>>> 
> TZ> * get Steinar, Dave, and others to try this out...
> TZ> * send salmon hunter drones after Lars to get him back on this :)
> TZ> * when a server is selected as the CLOUD-HOST, maybe save that
> TZ> choice? but I'm not sure if the choice should be persisted in
> TZ> `gnus-cloud-method' or in the newsrc.eld.
> TZ> * notice when the cloud data changes (maybe some IMAP magic can do it... but could just be done explicitly for now)
>>> 
>>> ...not done
>>> 
>>> I also changed some headers etc.
>
> EA> Hey I think I'm confused about what this is meant to do. My original
> EA> understanding was that this was for syncing NNTP marks between machines,
> EA> using an IMAP server. Is that what this is for?
>
> Yes. Plus files, optionally.
>
> EA> I have a laptop and a desktop. Both are running this branch, and on both
> EA> I've marked the same IMAP server as "cloud-sync".
>
> EA> But what happens now?
>
> Upload your first cloud pack from one machine and see if that works:
> `gnus-cloud-upload-data'

Oops, my misunderstanding was that I hadn't marked both a server-to-sync
and a server-to-keep-syncs. Once I'd done both "i" and "I", things
started going.

Initial bumps:

gnus-cloud-synced-files doesn't match up with my files (I'd recommend
the default use gnus-user-directory), and as the FIXME notes, I wasn't
able to redefine it to use a different (:directory...) syntax. I set it
directly to two files, and it was fine.

Weirdly, the (defvar gnus-cloud-sequence 1) didn't "work". The first
time I used it, gnus-cloud-sequence was nil, and so
`gnus-cloud-upload-data' threw an error in the call to `format'. I had
to explicitly setq the var to 1.

After that everything worked fine. I haven't yet gone to the office and
tried to download the data on the other machine.

Eric

Re: gnus-cloud work

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]

> After that everything worked fine. I haven't yet gone to the office and
> tried to download the data on the other machine.

Okay, now I've done that, and have got the sync IMAP group in both
places. Once again I had to manually setq the gnus-cloud-sequence
variable to make it work, and foolishly set it to 1 again, when I assume
I should have set it to 2? Or...? Anyhow, now I have two messages in the
sync group, both with :sequence 1. Evalling (gnus-cloud-download-data)
simply returns nil.

E

Re: gnus-cloud work

[Ted Zlatanov]

On Fri, 01 Jul 2016 10:00:29 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 

EA> Oops, my misunderstanding was that I hadn't marked both a server-to-sync
EA> and a server-to-keep-syncs. Once I'd done both "i" and "I", things
EA> started going.

Yeah, sorry about that. I'll write docs when the upload+download cycle
works :)

EA> Initial bumps:

EA> gnus-cloud-synced-files doesn't match up with my files (I'd recommend
EA> the default use gnus-user-directory), and as the FIXME notes, I wasn't
EA> able to redefine it to use a different (:directory...) syntax. I set it
EA> directly to two files, and it was fine.

I'm not sure if we should even allow relative filenames. Can you think
of a use case?

EA> Weirdly, the (defvar gnus-cloud-sequence 1) didn't "work". The first
EA> time I used it, gnus-cloud-sequence was nil, and so
EA> `gnus-cloud-upload-data' threw an error in the call to `format'. I had
EA> to explicitly setq the var to 1.

EA> After that everything worked fine. I haven't yet gone to the office and
EA> tried to download the data on the other machine.
...
EA> Okay, now I've done that, and have got the sync IMAP group in both
EA> places. Once again I had to manually setq the gnus-cloud-sequence
EA> variable to make it work, and foolishly set it to 1 again, when I assume
EA> I should have set it to 2? Or...? Anyhow, now I have two messages in the
EA> sync group, both with :sequence 1. Evalling (gnus-cloud-download-data)
EA> simply returns nil.

I think I've fixed all the places where `gnus-cloud-sequence' might be
nil with a single commit to the scratch branch.

The download will grab anything with a larger sequence:

    (dolist (header (gnus-cloud-available-chunks))
      (when (> (gnus-cloud-chunk-sequence (mail-header-subject header))
               (or sequence-override gnus-cloud-sequence -1))
    ...

So if you had seq 1 in IMAP, but locally you were at 2, nothing
happened as expected.

I think `gnus-cloud-available-chunks' will grow a bit as we find edge
cases. In this case (duplicate subject) it should pick the latest
article with that subject or error out, but not accept both. But that is
a pain to do in ELisp, whereas it may be easy at the IMAP level?

Ted

Re: gnus-cloud work

[Ted Zlatanov]

Update:

* newsrc data format changed, new entries have type :newsrc-data and the
  contents are inlined

* `M-x gnus-cloud-upload-all-data' will upload all the data, files and
  newsrc both

* (this must be run manually for now) `M-: (gnus-cloud-download-data t 1)'
  will get all the download data and install the newsrc and file data

Only full uploads/downloads work.

This is worth another test if people are adventurous, but *make sure you
back up your newsrc.eld* before downloading the data.

Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Right. You're going to have to use the Git Emacs checkout.

Yeah, looks like.  

Bummer! The threshold for that is higher (emacs with org-mode is an
important part of my daily workflow and I'm reluctant to risk that).

Re: gnus-cloud work

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> Right. You're going to have to use the Git Emacs checkout.

> Yeah, looks like.  

Ok, now I've built newest git emacs with your scratch/gnus-cloud branch
on a debian jessie computer.
  This one: https://steinar.bang.priv.no/2016/02/06/debian-jessie-on-intel-skylake/

I select my nnimap server in the server buffer, press 'i' to mark it as
gnus-cloud, and then arrow up one line and press 'I', and I still get
this error message:
Debugger entered--Lisp error: (error "Format specifier doesn’t match argument type")
  format("Subject: (sequence: %d type: %s storage-method: %s)\n" "UNKNOWN" :full epg)
  gnus-cloud-upload-data(t)
  gnus-server-toggle-cloud-method-server()
  funcall-interactively(gnus-server-toggle-cloud-method-server)
  call-interactively(gnus-server-toggle-cloud-method-server nil nil)
  command-execute(gnus-server-toggle-cloud-method-server)

Re: gnus-cloud work

[Ted Zlatanov]

On Sun, 03 Jul 2016 19:05:11 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Steinar Bang <sb@dod.no>:
>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>>> Right. You're going to have to use the Git Emacs checkout.

>> Yeah, looks like.  

SB> Ok, now I've built newest git emacs with your scratch/gnus-cloud branch
SB> on a debian jessie computer.
SB>   This one: https://steinar.bang.priv.no/2016/02/06/debian-jessie-on-intel-skylake/

Nice, all right.

SB> I select my nnimap server in the server buffer, press 'i' to mark it as
SB> gnus-cloud, and then arrow up one line and press 'I', and I still get
SB> this error message:
SB> Debugger entered--Lisp error: (error "Format specifier doesn’t match argument type")
SB>   format("Subject: (sequence: %d type: %s storage-method: %s)\n" "UNKNOWN" :full epg)
SB>   gnus-cloud-upload-data(t)
SB>   gnus-server-toggle-cloud-method-server()
SB>   funcall-interactively(gnus-server-toggle-cloud-method-server)
SB>   call-interactively(gnus-server-toggle-cloud-method-server nil nil)
SB>   command-execute(gnus-server-toggle-cloud-method-server)

My mistake. Fixed.

Ted

Re: gnus-cloud work

[Ted Zlatanov]

On Fri, 01 Jul 2016 21:00:27 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> Update:
TZ> * newsrc data format changed, new entries have type :newsrc-data and the
TZ>   contents are inlined

TZ> * `M-x gnus-cloud-upload-all-data' will upload all the data, files and
TZ>   newsrc both

TZ> * (this must be run manually for now) `M-: (gnus-cloud-download-data t 1)'
TZ>   will get all the download data and install the newsrc and file data

TZ> Only full uploads/downloads work.

TZ> This is worth another test if people are adventurous, but *make sure you
TZ> back up your newsrc.eld* before downloading the data.

I made a few more tweaks:

* only update if data is newer (can be overriden)
* add `gnus-cloud-interactive' variable to confirm updates (good for testing and paranoia)

The last piece missing is where to hook the upload and download code.

Upload could happen every 30 minutes or every time more than 10 groups
are updated? We have to be careful of race conditions where two machines
are not aware that the other has uploaded "locally newer" data. But
maybe that's OK, if we agree that the typical use case is one user on
multiple machines, not multiple users on multiple machines.

Download... not sure... maybe every time Gnus refreshes all groups with `g'?

Or maybe a background process could continuously monitor IMAP for new
articles in the cloud group? I don't know how that would work from the
Gnus side, but it would be super nice for the user. Any IMAP/nnimap experts?

Finally, currently the sequence number is a local number. Each machine
starts from 1 and walks up to the global max sequence (from the IMAP
articles). I think that's not ideal, because two machines can easily end
up uploading with the same sequence number without careful
synchronization. Global counters are hard to implement. Maybe it's
better to simply use a timestamp string or the raw IMAP article number
as the sequence number. The IMAP article number sequence in particular
is guaranteed by the IMAP server, so I'm leaning towards that. What do
you think?

Once upload, download, and sequence numbers are settled, I'll squash all
the commits and merge this into Emacs master branch.

Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> My mistake. Fixed.

Now when I first to 'i' and then 'I' on the nnimap servier, it says:
Uploading all data to Emacs Cloud with "nnimap:privat"

Then it prompts:
epg-encrypt-string:

When I C-g this (since I don't know what the epg-encrypt-string is
about), it says:
GPG error: "Encrypt failed", "Canceled; Exit"

Re: gnus-cloud work

[Ted Zlatanov]

On Wed, 06 Jul 2016 23:19:10 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> My mistake. Fixed.

SB> Now when I first to 'i' and then 'I' on the nnimap servier, it says:
SB> Uploading all data to Emacs Cloud with "nnimap:privat"

SB> Then it prompts:
SB> epg-encrypt-string:

SB> When I C-g this (since I don't know what the epg-encrypt-string is
SB> about), it says:
SB> GPG error: "Encrypt failed", "Canceled; Exit"

It's trying to encrypt the data it's about to upload. But the prompt
looks wrong. It looks OK for me, but I am using symmetric (just one
passphrase) encryption. So I'm not sure what's going on, this may need
further EPG setup.

Maybe Daiki Ueno has some idea? I'm probably setting up the EPG UI
incorrectly but I don't see many examples of that usage.

Ted

Re: gnus-cloud work

[Ted Zlatanov]

On Wed, 06 Jul 2016 10:50:17 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> Once upload, download, and sequence numbers are settled, I'll squash all
TZ> the commits and merge this into Emacs master branch.

I made upload and download manual processes for now. The sequence number is
still not settled but I thought it was ready for use so it's pushed now.

The Gnus manual has been updated with a new section explaining how to
use this and what settings can be tweaked.

Thanks
Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> I made upload and download manual processes for now. The sequence number is
> still not settled but I thought it was ready for use so it's pushed now.

> The Gnus manual has been updated with a new section explaining how to
> use this and what settings can be tweaked.

The scratch/gnus-cloud branch seems to be gone.  Have you merged your
changes to master?

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Once upload, download, and sequence numbers are settled, I'll squash
> all the commits and merge this into Emacs master branch.

Why squash the commits?

(I never do...)

Re: gnus-cloud work

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>> Once upload, download, and sequence numbers are settled, I'll squash
>> all the commits and merge this into Emacs master branch.

> Why squash the commits?

(because of the frequent merges from master which makes whatever history
is on the branch cluttered anyway, perhaps...?)

> (I never do...)

(maybe I would have in this case...? Depends on the ratio of real
commits to merge commits)

Re: gnus-cloud work

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>> I made upload and download manual processes for now. The sequence
>> number is still not settled but I thought it was ready for use so
>> it's pushed now.

>> The Gnus manual has been updated with a new section explaining how to
>> use this and what settings can be tweaked.

> The scratch/gnus-cloud branch seems to be gone.  Have you merged your
> changes to master?

Yes he has.
 http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=30b3a842ec87d27cfe003b6d4323689d48b3fcd2

I have added the modified files to the obsolete git Gnus (I don't have
the possibility to build git emacs while on vacation)
 https://github.com/steinarb/gnus-obsolete-gnus-cloud
and am trying to run this against emacs 24.4.

Since I now was actually able to read documentation I saw that I was
doing thing completely wrong (I did 'i' in the nnimap server followed by
'I' on the same server).

Now I did 'i' on each of the nntp servers I would like to sync followed
by I on the nnimap server, and this time it worked a lot better (but
still failed):
 Replication of nntp:news in the cloud will start
 Replication of nntp:news.gmane.org in the cloud will start
 Replication of news.eclipse.org in the cloud will start
 Uploading all data to Emacs Cloud server "nnimap:privat"
 gv-get: (epg-context-armor context) is not a valid place expression

What's epg-context-armor? Something emacs 25 specific? Or something I
can get support for on emacs 24.4?

I have the stack trace but it contained a lot of data so I won't include
it here.  I can put it in a gist if it has any interest...?

Re: gnus-cloud work

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> What's epg-context-armor? Something emacs 25 specific? Or something I
> can get support for on emacs 24.4?

epg-context-armor is a function available in emacs 24.4.

(But I have no idea why it fails... yet)

Re: gnus-cloud work

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Steinar Bang <sb@dod.no>:
>> What's epg-context-armor? Something emacs 25 specific? Or something I
>> can get support for on emacs 24.4?

> epg-context-armor is a function available in emacs 24.4.

I switched gnus-cloud-storage-method to base64+gzip and now storage
seems to work.  At least, the group nnimap+privat:Emacs-Cloud has been
created, containing 1 article.

Looking at the contents of the article (directly in the Maildir
directory of the folder on the dovecot server) it does indeed seem to be
base64 followed by gzip (ie. the body of the article is binary, meaning
gzip last).

That was a bit unexpected... I would have thought
gzip-followed-by-base64 would be the natural way to go for storing data
in an email system...?

Re: gnus-cloud work

[Ted Zlatanov]

On Sun, 24 Jul 2016 16:46:45 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> I switched gnus-cloud-storage-method to base64+gzip and now storage
SB> seems to work.  At least, the group nnimap+privat:Emacs-Cloud has been
SB> created, containing 1 article.

Yup.

SB> Looking at the contents of the article (directly in the Maildir
SB> directory of the folder on the dovecot server) it does indeed seem to be
SB> base64 followed by gzip (ie. the body of the article is binary, meaning
SB> gzip last).

SB> That was a bit unexpected... I would have thought
SB> gzip-followed-by-base64 would be the natural way to go for storing data
SB> in an email system...?

I don't know why that would happen. Look at the code:

#+begin_src lisp
(defun gnus-cloud-encode-data ()
  (cond
   ((eq gnus-cloud-storage-method 'base64-gzip)
    (call-process-region (point-min) (point-max) "gzip"
                         t (current-buffer) nil
                         "-c"))

   ((memq gnus-cloud-storage-method '(base64 base64-gzip))
    (base64-encode-region (point-min) (point-max)))
...
#+end_src

It gzips, then does base64.

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:

>> Once upload, download, and sequence numbers are settled, I'll squash
>> all the commits and merge this into Emacs master branch.

SB> Why squash the commits?

I made a lot of small commits playing around with the rewrite. Most had
bad commit messages and were unstructured (touching many areas). Also,
this is a pretty big rewrite, so it makes sense to squash into one large
change that can be reviewed easily. But I certainly could have done
small commits too, I was just lazy :)

Ted

Re: gnus-cloud work

[Andreas Schwab]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I don't know why that would happen. Look at the code:
>
> #+begin_src lisp
> (defun gnus-cloud-encode-data ()
>   (cond
>    ((eq gnus-cloud-storage-method 'base64-gzip)
>     (call-process-region (point-min) (point-max) "gzip"
>                          t (current-buffer) nil
>                          "-c"))
>
>    ((memq gnus-cloud-storage-method '(base64 base64-gzip))
>     (base64-encode-region (point-min) (point-max)))
> ...
> #+end_src
>
> It gzips, then does base64.

Only one of the two branches is executed.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: gnus-cloud work

[Ted Zlatanov]

On Mon, 25 Jul 2016 19:42:52 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Only one of the two branches is executed.

Thanks, I couldn't see it for some reason!

Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> On Mon, 25 Jul 2016 19:42:52 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Only one of the two branches is executed.

> Thanks, I couldn't see it for some reason!

The results looks more as expected, the body of the message is
base64-encoded, and the size is increased with approximately one third:
  /home/sb/Maildir/.Emacs-Cloud/new:
  total used in directory 272 available 13135812
  drwx------ 2 sb sb  4096 Jul 26 08:24 .
  drwx------ 5 sb sb  4096 Jul 26 08:24 ..
  -rw------- 1 sb sb 60936 Jul 24 14:49 1469368177.M704006P32710.someserver.somedomain.com,S=60936,W=61154
  -rw------- 1 sb sb 95517 Jul 26 08:24 1469517852.M388173P8614.someserver.somedomain.com,S=95517,W=96761
  -rw------- 1 sb sb 95616 Jul 26 08:24 1469517888.M603268P8614.someserver.somedomain.com,S=95616,W=96861

Should I delete the old uploads (the first 60k sized, and the second
where I forgot two servers) before proceeding, or don't they matter?

I noticed that the (cloud-sync) and (CLOUD-HOST) marks were gone from
the Server buffer after the start of a new gnus in a fresh emacs.  Are
the server buffer marks supposed to be persistent? Or are they just
there for the first upload?

Re: gnus-cloud work

[Ted Zlatanov]

On Tue, 26 Jul 2016 10:15:29 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> On Mon, 25 Jul 2016 19:42:52 +0200 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Only one of the two branches is executed.

>> Thanks, I couldn't see it for some reason!

SB> The results looks more as expected, the body of the message is
SB> base64-encoded, and the size is increased with approximately one third:
SB>   /home/sb/Maildir/.Emacs-Cloud/new:
SB>   total used in directory 272 available 13135812
SB>   drwx------ 2 sb sb  4096 Jul 26 08:24 .
SB>   drwx------ 5 sb sb  4096 Jul 26 08:24 ..
SB>   -rw------- 1 sb sb 60936 Jul 24 14:49 1469368177.M704006P32710.someserver.somedomain.com,S=60936,W=61154
SB>   -rw------- 1 sb sb 95517 Jul 26 08:24 1469517852.M388173P8614.someserver.somedomain.com,S=95517,W=96761
SB>   -rw------- 1 sb sb 95616 Jul 26 08:24 1469517888.M603268P8614.someserver.somedomain.com,S=95616,W=96861

SB> Should I delete the old uploads (the first 60k sized, and the second
SB> where I forgot two servers) before proceeding, or don't they matter?

SB> I noticed that the (cloud-sync) and (CLOUD-HOST) marks were gone from
SB> the Server buffer after the start of a new gnus in a fresh emacs.  Are
SB> the server buffer marks supposed to be persistent? Or are they just
SB> there for the first upload?

Both should be persistent.

The (cloud-sync) flag is stored with `gnus-cloud-covered-servers'. It
should be in your newsrc.eld. We can make it a defcustom or a server
parameter, I'm not sure which is more appropriate. It feels like a
defcustom.

The (CLOUD-HOST) flag is stored with `gnus-cloud-method'. It should be
set as a defcustom, can you check if it was saved? I didn't save it
explicitly, just set it in gnus-srvr.el:

    (custom-set-variables '(gnus-cloud-method server))

So maybe I need to save the customization? I don't see a way to do it
just for that one variable, unfortunately. Maybe it should be a prompt
when you hit `I' to save all customizations and run
`custom-save-variables'? That's the only way I can see.

Thanks
Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Both should be persistent.

They now seem to be.  After I saved the new initial data, I stopped
gnus, created a new emacs and started Gnus in the new emacs, and checked
the server buffer and this time (cloud-sync) and (CLOUD-HOST) were
present.

I don't know what happened the first time.

Sorry about the noise!

Re: gnus-cloud work

[Steinar Bang]

Running gnus-clould-upload-all-data gives me the error:
 gnus-get-function: No such function: nntp-request-create-group

Note: I'm running obsolete-gnus on emacs 24.4 with the gnus-cloud
changes applied, so it may well be that I don't have this method.

Here's my obsolete-gnus for those that may be interested:
 https://github.com/steinarb/gnus-obsolete-gnus-cloud

Re: gnus-cloud work

[Ted Zlatanov]

On Tue, 26 Jul 2016 20:55:59 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> Running gnus-clould-upload-all-data gives me the error:
SB>  gnus-get-function: No such function: nntp-request-create-group

SB> Note: I'm running obsolete-gnus on emacs 24.4 with the gnus-cloud
SB> changes applied, so it may well be that I don't have this method.

SB> Here's my obsolete-gnus for those that may be interested:
SB>  https://github.com/steinarb/gnus-obsolete-gnus-cloud

That seems like a bug or misconfiguration, it's trying to create the
cloud storage group on a NNTP server.

What's your `gnus-cloud-method'? Is it an NNTP or an IMAP server?

Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> That seems like a bug or misconfiguration, it's trying to create the
> cloud storage group on a NNTP server.

> What's your `gnus-cloud-method'? Is it an NNTP or an IMAP server?

Misconfiguration, probably.

'C-h v' says that gnus-cloud-method is nil.

Let me see what the documentation says...? Ah, gnus-cloud-method was
supposed to have been set by the upload using the customize facilities.

The variable *was* set by the upload, but perhaps not to something
useful...?  This is what it was set to (from ~/.emacs):
 (custom-set-variables
  ...
  '(gnus-cloud-method server)
  '(gnus-cloud-storage-method (quote base64-gzip))
  ...

(the gnus-cloud-storage-method was explicitly customized by me, since
the EPG stuff didn't work)

Re: gnus-cloud work

[Ted Zlatanov]

On Tue, 26 Jul 2016 23:19:38 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> That seems like a bug or misconfiguration, it's trying to create the
>> cloud storage group on a NNTP server.

>> What's your `gnus-cloud-method'? Is it an NNTP or an IMAP server?

SB> Misconfiguration, probably.

SB> 'C-h v' says that gnus-cloud-method is nil.

SB> Let me see what the documentation says...? Ah, gnus-cloud-method was
SB> supposed to have been set by the upload using the customize facilities.

SB> The variable *was* set by the upload, but perhaps not to something
SB> useful...?  This is what it was set to (from ~/.emacs):
SB>  (custom-set-variables
SB>   ...
SB>   '(gnus-cloud-method server)

Use `I' again to set it from the Server buffer. Then do `M-x
custom-save-all'.

I don't know if the save should be automatic. Here's how I think it
could work, by asking the user. What do you think?

Ted

diff --git a/lisp/gnus/gnus-srvr.el b/lisp/gnus/gnus-srvr.el
index 66fb9ee..e83deeb 100644
--- a/lisp/gnus/gnus-srvr.el
+++ b/lisp/gnus/gnus-srvr.el
@@ -1156,7 +1156,13 @@ gnus-server-toggle-cloud-method-server
     (unless (gnus-cloud-host-acceptable-method-p server)
       (error "The server under point can't host the Emacs Cloud"))
 
-    (custom-set-variables '(gnus-cloud-method server))
+    (when (not (string-equal gnus-cloud-method server))
+      (custom-set-variables '(gnus-cloud-method server))
+      ;; Note we can't use `Custom-save' here.
+      (when (gnus-yes-or-no-p
+             (format "The new cloud host server is %S now. Save your customizations (note it may save other customizations)? "
+                     server))
+        (custom-save-variables)))
     (when (gnus-yes-or-no-p (format "Upload Cloud data to %S now? " server))
       (gnus-message 1 "Uploading all data to Emacs Cloud server %S" server)
       (gnus-cloud-upload-data t))))

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Use `I' again to set it from the Server buffer. Then do `M-x
> custom-save-all'.

Emacs 24.4 doesn't have custom-save-all.

I have tried
 'M-x customize-save-customized'
and that did nothing.

'M-x customize-save-variable' and then give 'gnus-cloud-method' at the
prompt saved this setting in .emacs:
 '(gnus-cloud-method nil)

(which probably isn't right either...?)

> I don't know if the save should be automatic. Here's how I think it
> could work, by asking the user. What do you think?

Looks good to me.

Re: gnus-cloud work

[Ted Zlatanov]

On Wed, 27 Jul 2016 17:59:56 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> Use `I' again to set it from the Server buffer. Then do `M-x
>> custom-save-all'.

SB> Emacs 24.4 doesn't have custom-save-all.

SB> I have tried
SB>  'M-x customize-save-customized'
SB> and that did nothing.

SB> 'M-x customize-save-variable' and then give 'gnus-cloud-method' at the
SB> prompt saved this setting in .emacs:
SB>  '(gnus-cloud-method nil)

SB> (which probably isn't right either...?)

The newly pushed code should do the right thing.

Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> The newly pushed code should do the right thing.

Ok, now the gnus-cloud-method is saved like this in .emacs:
 '(gnus-cloud-method "nnimap:privat")
which looks a lot better.

I tried '~ RET' to save new data, and got the error below.

(it could be that this is caused by that there were no new read marks
after the 'I' upload...?)

Hm... I see they are related to the gnus.spam.reported group, which gas
some Summary buffer modifications.  The group has the following added to
the group parameters:
 (gnus-summary-line-format "%U%uS: %uT %-40,40s %[%-25,25uG %-20,20uA%z%I%(%-23,23f%]%)\n"))

Here's the stack trace from the error:


Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  string-match("gmane.spam.reported" nil)
  (if (string-match "gmane.spam.reported" gnus-newsgroup-name) (progn (save-excursion (let ((queue nil) (score 0) line) (goto-char (point-min)) (while (search-forward "\nX-Gmane-Queue:" nil t) (setq line (split-string (buffer-substring ... ...))) (message "Resolving %s" (nth 4 line)) (setq queue (cons (list ... ... ... ...) queue))) (goto-char (point-max)) (if (search-backward "\nX-Spam-Report:" nil t) (progn (goto-char (match-end 0)) (setq score (read ...)) (if (numberp score) nil (setq score 0)))) (mail-header-set-extra header (list (list (quote Id) (nth 1 ...) (nth 3 ...)) (list (quote Ip) (nth 4 line)) (list (quote Score) score) (cons (quote Queue) (spam-gmane-collate-reports queue))))))))
  spam-gmane-massage-header([1 "(sequence: UNKNOWN type: :full storage-method: base64-gzip)" "nobody@gnus.cloud.invalid" "" "<86h9bff8nh.fsf@totally-fudged-out-message-id>" nil 61154 214 nil nil])
  gnus-get-newsgroup-headers([0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ...])
  nnvirtual-convert-headers()
  gnus-agent-retrieve-headers((1 2 3 4 5 6 7 8) "nnimap+privat:Emacs-Cloud" nil)
  gnus-retrieve-headers((1 2 3 4 5 6 7 8) "nnimap+privat:Emacs-Cloud" nil)
  gnus-cache-retrieve-headers((1 2 3 4 5 6 7 8) "nnimap+privat:Emacs-Cloud" nil)
  gnus-retrieve-headers((1 2 3 4 5 6 7 8) "nnimap+privat:Emacs-Cloud")
  (if (gnus-retrieve-headers (gnus-uncompress-range active) group) (progn (save-current-buffer (set-buffer nntp-server-buffer) (goto-char (point-min)) (while (and (not (eobp)) (setq head (nnheader-parse-head))) (setq headers (cons head headers))))))
  (let* ((group (gnus-group-full-name gnus-cloud-group-name gnus-cloud-method)) (active (symbol-value (intern-soft group gnus-active-hashtb))) headers head) (if (gnus-retrieve-headers (gnus-uncompress-range active) group) (progn (save-current-buffer (set-buffer nntp-server-buffer) (goto-char (point-min)) (while (and (not (eobp)) (setq head (nnheader-parse-head))) (setq headers (cons head headers)))))) (sort (nreverse headers) (function (lambda (h1 h2) (> (gnus-cloud-chunk-sequence (aref h1 1)) (gnus-cloud-chunk-sequence (aref h2 1)))))))
  gnus-cloud-available-chunks()
  (let ((--dolist-tail-- (gnus-cloud-available-chunks)) header) (while --dolist-tail-- (setq header (car --dolist-tail--)) (if (> (gnus-cloud-chunk-sequence (aref header 1)) (or sequence-override gnus-cloud-sequence -1)) (progn (if (string-match (format "storage-method: %s" gnus-cloud-storage-method) (aref header 1)) (setq articles (cons (aref header 0) articles)) (gnus-message 1 "Skipping article %s because it didn't match the Gnus Cloud method %s: %s" (aref header 0) gnus-cloud-storage-method (aref header 1))))) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((articles nil) chunks) (let ((--dolist-tail-- (gnus-cloud-available-chunks)) header) (while --dolist-tail-- (setq header (car --dolist-tail--)) (if (> (gnus-cloud-chunk-sequence (aref header 1)) (or sequence-override gnus-cloud-sequence -1)) (progn (if (string-match (format "storage-method: %s" gnus-cloud-storage-method) (aref header 1)) (setq articles (cons ... articles)) (gnus-message 1 "Skipping article %s because it didn't match the Gnus Cloud method %s: %s" (aref header 0) gnus-cloud-storage-method (aref header 1))))) (setq --dolist-tail-- (cdr --dolist-tail--)))) (if articles (progn (nnimap-request-articles (nreverse articles) gnus-cloud-group-name) (save-current-buffer (set-buffer nntp-server-buffer) (goto-char (point-min)) (while (re-search-forward "^Gnus-Cloud-Version " ni
 l t) (beginning-of-line) (setq chunks (cons (gnus-cloud-parse-chunk) chunks)) (forward-line 1))))) (if update (mapcar (function gnus-cloud-update-all) chunks) chunks))
  gnus-cloud-download-data(t)
  gnus-cloud-download-all-data()
  call-interactively(gnus-cloud-download-all-data nil nil)
  command-execute(gnus-cloud-download-all-data)

Re: gnus-cloud work

[Ted Zlatanov]

On Thu, 28 Jul 2016 19:18:22 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> The newly pushed code should do the right thing.

SB> Ok, now the gnus-cloud-method is saved like this in .emacs:
SB>  '(gnus-cloud-method "nnimap:privat")
SB> which looks a lot better.

Cool.

SB> I tried '~ RET' to save new data, and got the error below.

SB> (it could be that this is caused by that there were no new read marks
SB> after the 'I' upload...?)

SB> Hm... I see they are related to the gnus.spam.reported group, which gas
SB> some Summary buffer modifications.  The group has the following added to
SB> the group parameters:
SB>  (gnus-summary-line-format "%U%uS: %uT %-40,40s %[%-25,25uG %-20,20uA%z%I%(%-23,23f%]%)\n"))

I think the error is in `spam-gmane-massage-header' but I'm not sure why
it's called. I don't have that function, is it an external package?

SB> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
SB>   string-match("gmane.spam.reported" nil)
SB>   (if (string-match "gmane.spam.reported" gnus-newsgroup-name) (progn (save-excursion (let ((queue nil) (score 0) line) (goto-char (point-min)) (while (search-forward "\nX-Gmane-Queue:" nil t) (setq line (split-string (buffer-substring ... ...))) (message "Resolving %s" (nth 4 line)) (setq queue (cons (list ... ... ... ...) queue))) (goto-char (point-max)) (if (search-backward "\nX-Spam-Report:" nil t) (progn (goto-char (match-end 0)) (setq score (read ...)) (if (numberp score) nil (setq score 0)))) (mail-header-set-extra header (list (list (quote Id) (nth 1 ...) (nth 3 ...)) (list (quote Ip) (nth 4 line)) (list (quote Score) score) (cons (quote Queue) (spam-gmane-collate-reports queue))))))))
SB>   spam-gmane-massage-header([1 "(sequence: UNKNOWN type: :full storage-method: base64-gzip)" "nobody@gnus.cloud.invalid" "" "<86h9bff8nh.fsf@totally-fudged-out-message-id>" nil 61154 214 nil nil])
SB>   gnus-get-newsgroup-headers([0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ...])
SB>   nnvirtual-convert-headers()
SB>   gnus-agent-retrieve-headers((1 2 3 4 5 6 7 8) "nnimap+privat:Emacs-Cloud" nil)

Something in that call chain is breaking when gnus-cloud sets the marks,
then refreshes the group. It looks like the code expects
`gnus-newsgroup-name' to be set but in this case (when we refresh
without entering the group) it isn't. I don't know what's the exact
situation, this is just a guess, sorry.

Ted

Re: gnus-cloud work

[Steinar Bang]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> I think the error is in `spam-gmane-massage-header' but I'm not sure why
> it's called. I don't have that function, is it an external package?

Attached at the end of this message.

FWIW It has two gnus-sync-related changes in its history:
 commit 24803a3cca2d66b03681536ea58cc826eb64dc55
 Author: Steinar Bang <sb@dod.no>
 Date:   Thu Nov 17 21:55:51 2011 +0100

    Corrections to Co-exist with gnus-sync.el v2: end let clause correctly, and set url-request-data as well

 commit ffba929dc12a8a792b1e327817c986a3a4010751
 Author: Steinar Bang <sb@dod.no>
 Date:   Mon Nov 7 23:58:50 2011 +0100

    Co-exist with gnus-sync.el v2: don't fail because of a void url-request-method.

(The author isn't significant, the file is just versioned as a part of
my home directory version control)

> Something in that call chain is breaking when gnus-cloud sets the
> marks, then refreshes the group. It looks like the code expects
> `gnus-newsgroup-name' to be set but in this case (when we refresh
> without entering the group) it isn't. I don't know what's the exact
> situation, this is just a guess, sorry.

Hm... FWIW the group can be synced with gnus-sync (I've done so for
years).


[-- Attachment #2: Gmane spam approval support for Gnus --]
[-- Type: application/emacs-lisp, Size: 18330 bytes --]

Re: gnus-cloud work

[Steinar Bang]

With all that has happened with gmane lately it may be that there is no
more need for the spam approval interface, and that there is no need for
synching the groups.

So I've ditched ger.gmane.org from the set of synched servers, and now
upload and download seems to work.

Ie. I don't know if it works yet, because I don't have any other gnusen
to sync with while on vacation.  We'll see if I can use this to sync
read articles when I get home.

How do you plan to handle Emacs-Cloud cleanup? With 10 messages in it,
my folder is using sligthly less than 1MB of disk... not much, but with
frequent saves that could add up.

What happenend Gmane? (Was: Re: gnus-cloud work)

[Byung-Hee HWANG (황병희)]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

On 2016년 7월 30일 오전 5시 35분 26초 GMT+09:00, Steinar Bang <sb@dod.no> wrote:
>With all that has happened with gmane lately it may be that there is no
>more need for the spam approval interface, and that there is no need
>for
>synching the groups.
>
>So I've ditched ger.gmane.org from the set of synched servers, and now
>upload and download seems to work.
>
>Ie. I don't know if it works yet, because I don't have any other gnusen
>to sync with while on vacation.  We'll see if I can use this to sync
>read articles when I get home.
>
>How do you plan to handle Emacs-Cloud cleanup? With 10 messages in it,
>my folder is using sligthly less than 1MB of disk... not much, but with
>frequent saves that could add up.

When i connect android smartphone, chrome gave me wonderful screenshot (attached file). How is it going gmane?
 
Sincerely,

ps. And yes, i like gnus' cloud!!! 
pps. Cheer up !!!
-- 
^고맙습니다 감사합니다_^))//

[-- Attachment #2: Screenshot_2016-07-30-07-57-45.png --]
[-- Type: image/png, Size: 127126 bytes --]

Re: What happenend Gmane?

[Steinar Bang]

>>>>> "Byung-Hee HWANG (황병희)" <soyeomul@doraji.xyz>:

> When i connect android smartphone, chrome gave me wonderful screenshot
> (attached file). How is it going gmane?

I haven't looked at the screenshot, but here is what I've seen:
 https://lars.ingebrigtsen.no/2016/07/28/the-end-of-gmane/
 https://lwn.net/Articles/695695/
 https://developers.slashdot.org/story/16/07/28/2059249/the-end-of-gmane
 https://news.ycombinator.com/item?id=12180547

Re: What happenend Gmane?

[Byung-Hee HWANG (황병희)]

On 2016년 7월 30일 오후 3시 36분 12초 GMT+09:00, Steinar Bang <sb@dod.no> wrote:
>>>>>> "Byung-Hee HWANG (황병희)" <soyeomul@doraji.xyz>:
>
>> When i connect android smartphone, chrome gave me wonderful
>screenshot
>> (attached file). How is it going gmane?
>
>I haven't looked at the screenshot, but here is what I've seen:
> https://lars.ingebrigtsen.no/2016/07/28/the-end-of-gmane/
> https://lwn.net/Articles/695695/
>https://developers.slashdot.org/story/16/07/28/2059249/the-end-of-gmane
> https://news.ycombinator.com/item?id=12180547

Now i read the big news.

And Sorry for off topic, i really like Gmane very much. Please  tell Lars "Hey cheer up Lars You are my hero for open source world!"

Sincerely, 

-- 
^고맙습니다 감사합니다_^))//

Re: gnus-cloud work

[Ted Zlatanov]

On Fri, 29 Jul 2016 22:35:26 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> How do you plan to handle Emacs-Cloud cleanup? With 10 messages in it,
SB> my folder is using sligthly less than 1MB of disk... not much, but with
SB> frequent saves that could add up.

It's not really a concern, but you can set up Gnus total expiry if you
are worried.

Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> On Fri, 29 Jul 2016 22:35:26 +0200 Steinar Bang <sb@dod.no> wrote: 
SB> How do you plan to handle Emacs-Cloud cleanup? With 10 messages in it,
SB> my folder is using sligthly less than 1MB of disk... not much, but with
SB> frequent saves that could add up.

> It's not really a concern, but you can set up Gnus total expiry if you
> are worried.

What should the expiration criteria be?

Re: gnus-cloud work

[Ted Zlatanov]

On Mon, 01 Aug 2016 21:34:30 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> On Fri, 29 Jul 2016 22:35:26 +0200 Steinar Bang <sb@dod.no> wrote: 
SB> How do you plan to handle Emacs-Cloud cleanup? With 10 messages in it,
SB> my folder is using sligthly less than 1MB of disk... not much, but with
SB> frequent saves that could add up.

>> It's not really a concern, but you can set up Gnus total expiry if you
>> are worried.

SB> What should the expiration criteria be?

Currently gnus-cloud.el only does full saves.

Like any backup, retention policy depends on frequency, cost of storage,
and the value of the data. I don't know any of those things for you. As
a guess, if you save every day, data older than 2 weeks is probably not
very useful and can be discarded.

Once we have incremental saves, you can again figure out how far back
your data needs to go.

Ted

Re: gnus-cloud work

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Once we have incremental saves, you can again figure out how far back
> your data needs to go.

Ah, you have planned to do incremental saves?

No worries from my side then.  :-)

I figure I won't do anything on the expiry side until that is in place.

gnus-cloud: Invalid base64 data (Was: gnus-cloud work)

[Steinar Bang]

I'm back from vacation and I tried my first real '~ d' on a different
emacs today, and got the message:
 Invalid base64 data

Since the EPG stuff didn't work for me, I have the following
customization valur in .emacs (including the .emacs on the computer I'm
trying to do the download on):
 '(gnus-cloud-storage-method (quote base64-gzip))

I have checked the actual message in the Maildir folder on the computer
I'm running Dovecot on, and its body looks to be valid base64, including
padding with two trailing "==".  The base64 part of the body also
decodes with no error message using the base64-decode-region function.

However, the start of the message looks like this:

Gnus-Cloud-Version 1
H4sIAGKcoVcAC8Q8a3PbtrKfk5n+BwzvnbHkhIok24mjnHtjx0kfc1qnU/u0nbHdHoiEJNYkoBCg
...


Could it be that "Gnus-Cloud-Version 1"+RET is part of the stuff that is
tried to be decoded as base64?

Here's the stack trace from the error:
Debugger entered--Lisp error: (error "Invalid base64 data")
  base64-decode-region(1 986312)
  (cond ((memq gnus-cloud-storage-method (quote (base64 base64-gzip))) (base64-decode-region (point-min) (point-max))) ((eq gnus-cloud-storage-method (quote base64-gzip)) (call-process-region (point-min) (point-max) "gunzip" t (current-buffer) nil "-c")) ((eq gnus-cloud-storage-method (quote epg)) (let* ((context (epg-make-context (quote OpenPGP))) (data (epg-decrypt-string context (buffer-substring-no-properties (point-min) (point-max))))) (delete-region (point-min) (point-max)) (insert data))) ((null gnus-cloud-storage-method) (gnus-message 5 "Reading cloud data as plaintext")) (t (gnus-error 1 "Invalid cloud storage method %S" gnus-cloud-storage-method)))
  gnus-cloud-decode-data()
  (cond ((= version 1) (gnus-cloud-decode-data) (goto-char (point-min)) (gnus-cloud-parse-version-1)) (t (error "Unsupported Cloud chunk version %s" version)))
  (progn (mm-disable-multibyte) (insert data) (cond ((= version 1) (gnus-cloud-decode-data) (goto-char (point-min)) (gnus-cloud-parse-version-1)) (t (error "Unsupported Cloud chunk version %s" version))))
  (unwind-protect (progn (mm-disable-multibyte) (insert data) (cond ((= version 1) (gnus-cloud-decode-data) (goto-char (point-min)) (gnus-cloud-parse-version-1)) (t (error "Unsupported Cloud chunk version %s" version)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (mm-disable-multibyte) (insert data) (cond ((= version 1) (gnus-cloud-decode-data) (goto-char (point-min)) (gnus-cloud-parse-version-1)) (t (error "Unsupported Cloud chunk version %s" version)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (mm-disable-multibyte) (insert data) (cond ((= version 1) (gnus-cloud-decode-data) (goto-char (point-min)) (gnus-cloud-parse-version-1)) (t (error "Unsupported Cloud chunk version %s" version)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  (let ((version (string-to-number (match-string 1))) (data (buffer-substring (point) (point-max)))) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (mm-disable-multibyte) (insert data) (cond ((= version 1) (gnus-cloud-decode-data) (goto-char ...) (gnus-cloud-parse-version-1)) (t (error "Unsupported Cloud chunk version %s" version)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))))
  (save-excursion (if (looking-at "Gnus-Cloud-Version \\([0-9]+\\)") nil (error "Not a valid Cloud chunk in the current buffer")) (forward-line 1) (let ((version (string-to-number (match-string 1))) (data (buffer-substring (point) (point-max)))) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (mm-disable-multibyte) (insert data) (cond (... ... ... ...) (t ...))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))))
  gnus-cloud-parse-chunk()
  (cons (gnus-cloud-parse-chunk) chunks)
  (setq chunks (cons (gnus-cloud-parse-chunk) chunks))
  (while (re-search-forward "^Gnus-Cloud-Version " nil t) (beginning-of-line) (setq chunks (cons (gnus-cloud-parse-chunk) chunks)) (forward-line 1))
  (save-current-buffer (set-buffer nntp-server-buffer) (goto-char (point-min)) (while (re-search-forward "^Gnus-Cloud-Version " nil t) (beginning-of-line) (setq chunks (cons (gnus-cloud-parse-chunk) chunks)) (forward-line 1)))
  (progn (nnimap-request-articles (nreverse articles) gnus-cloud-group-name) (save-current-buffer (set-buffer nntp-server-buffer) (goto-char (point-min)) (while (re-search-forward "^Gnus-Cloud-Version " nil t) (beginning-of-line) (setq chunks (cons (gnus-cloud-parse-chunk) chunks)) (forward-line 1))))
  (if articles (progn (nnimap-request-articles (nreverse articles) gnus-cloud-group-name) (save-current-buffer (set-buffer nntp-server-buffer) (goto-char (point-min)) (while (re-search-forward "^Gnus-Cloud-Version " nil t) (beginning-of-line) (setq chunks (cons (gnus-cloud-parse-chunk) chunks)) (forward-line 1)))))
  (let ((articles nil) chunks) (let ((--dolist-tail-- (gnus-cloud-available-chunks)) header) (while --dolist-tail-- (setq header (car --dolist-tail--)) (if (> (gnus-cloud-chunk-sequence (aref header 1)) (or sequence-override gnus-cloud-sequence -1)) (progn (if (string-match (format "storage-method: %s" gnus-cloud-storage-method) (aref header 1)) (setq articles (cons ... articles)) (gnus-message 1 "Skipping article %s because it didn't match the Gnus Cloud method %s: %s" (aref header 0) gnus-cloud-storage-method (aref header 1))))) (setq --dolist-tail-- (cdr --dolist-tail--)))) (if articles (progn (nnimap-request-articles (nreverse articles) gnus-cloud-group-name) (save-current-buffer (set-buffer nntp-server-buffer) (goto-char (point-min)) (while (re-search-forward "^Gnus-Cloud-Version " ni
 l t) (beginning-of-line) (setq chunks (cons (gnus-cloud-parse-chunk) chunks)) (forward-line 1))))) (if update (mapcar (function gnus-cloud-update-all) chunks) chunks))
  gnus-cloud-download-data(t)
  gnus-cloud-download-all-data()
  call-interactively(gnus-cloud-download-all-data nil nil)
  command-execute(gnus-cloud-download-all-data)

Re: gnus-cloud work

[Dave Abrahams]

on Mon Aug 01 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Fri, 29 Jul 2016 22:35:26 +0200 Steinar Bang <sb@dod.no> wrote: 
>
> SB> How do you plan to handle Emacs-Cloud cleanup? With 10 messages in it,
> SB> my folder is using sligthly less than 1MB of disk... not much, but with
> SB> frequent saves that could add up.
>
> It's not really a concern, but you can set up Gnus total expiry if you
> are worried.


Hi all,

Is emacs-25.1 recent enough to have the latest gnus-cloud support?
What are the steps to try it out?

Many thanks,

-- 
-Dave

Re: gnus-cloud work

[Ted Zlatanov]

On Sun, 18 Sep 2016 17:06:32 -0700 Dave Abrahams <dave@boostpro.com> wrote: 

DA> on Mon Aug 01 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

>> On Fri, 29 Jul 2016 22:35:26 +0200 Steinar Bang <sb@dod.no> wrote: 
>> 
SB> How do you plan to handle Emacs-Cloud cleanup? With 10 messages in it,
SB> my folder is using sligthly less than 1MB of disk... not much, but with
SB> frequent saves that could add up.
>> 
>> It's not really a concern, but you can set up Gnus total expiry if you
>> are worried.

DA> Is emacs-25.1 recent enough to have the latest gnus-cloud support?

Gnus is part of the Emacs source code. But the work I did in July was
too late for the 25.1 release (yes, that's a pain). You should just
check out and compile the latest master, it's really easy.

DA> What are the steps to try it out?

The docs have been added:

(info "(gnus) The Gnus Cloud")

Ted

Re: gnus-cloud work

[Dave Abrahams]

on Tue Sep 20 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Sun, 18 Sep 2016 17:06:32 -0700 Dave Abrahams <dave@boostpro.com> wrote: 
>
> DA> on Mon Aug 01 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:
>
>>> On Fri, 29 Jul 2016 22:35:26 +0200 Steinar Bang <sb@dod.no> wrote: 
>>> 
> SB> How do you plan to handle Emacs-Cloud cleanup? With 10 messages in it,
> SB> my folder is using sligthly less than 1MB of disk... not much, but with
> SB> frequent saves that could add up.
>>> 
>>> It's not really a concern, but you can set up Gnus total expiry if you
>>> are worried.
>
> DA> Is emacs-25.1 recent enough to have the latest gnus-cloud support?
>
> Gnus is part of the Emacs source code. But the work I did in July was
> too late for the 25.1 release (yes, that's a pain). You should just
> check out and compile the latest master, it's really easy.
>
> DA> What are the steps to try it out?
>
> The docs have been added:
>
> (info "(gnus) The Gnus Cloud")

Hi Ted,

I'm having two problems, now: first, every time I try to upload or
download with gnus-cloud, GPG is asking me for a passphrase, and even
when I tick the “save in keychain” checkbox I'm offered, it keeps
asking, every time.  'Scuse my GPG naïveté, but is there a way to get
this to stop?

Second, it doesn't seem to have any effect.  I have 44 unread messages
on this machine from gmane.comp.lang.swift.evolution.  When I upload
from here and then download to another machine, I still have 167 unread
messages in that group.

Lastly, I have some questions: 

a. what's the effect supposed to be if I forget to upload the data and
   do some reading elsewhere?  Is there a way of merging marks, or...?

b. how much of my Gnus configuration is supposed to get saved?  I'd
   kinda rather keep everything.

c. does gnus-cloud handle something that wouldn't be covered by keeping
   my entire ~/News and ~/Mail directories cloud-synced by other means?

Thanks,

-- 
-Dave

Re: gnus-cloud work

[Dave Abrahams]

on Tue Sep 20 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Sun, 18 Sep 2016 17:06:32 -0700 Dave Abrahams <dave@boostpro.com> wrote: 
>
> DA> on Mon Aug 01 2016, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:
>
>>> On Fri, 29 Jul 2016 22:35:26 +0200 Steinar Bang <sb@dod.no> wrote: 
>>> 
> SB> How do you plan to handle Emacs-Cloud cleanup? With 10 messages in it,
> SB> my folder is using sligthly less than 1MB of disk... not much, but with
> SB> frequent saves that could add up.
>>> 
>>> It's not really a concern, but you can set up Gnus total expiry if you
>>> are worried.
>
> DA> Is emacs-25.1 recent enough to have the latest gnus-cloud support?
>
> Gnus is part of the Emacs source code. But the work I did in July was
> too late for the 25.1 release (yes, that's a pain). You should just
> check out and compile the latest master, it's really easy.
>
> DA> What are the steps to try it out?
>
> The docs have been added:
>
> (info "(gnus) The Gnus Cloud")

Nice; after a few inexplicable glitches I got it set up!  I still need to try
Gnus on another machine to observe the seamless continuity... :-)

One thing I notice now is that `G' in my *Group* buffer now takes a long
time and sometimes appears to truly hang here:

Debugger entered: ("Quit")
  nnimap-wait-for-response(167)
  nnimap-request-list("imap.gmail.com")
  gnus-request-list((nnimap "imap.gmail.com" (nnimap-server-port 993) (nnimap-stream ssl)))
  gnus-read-active-file-1((nnimap "imap.gmail.com" (nnimap-server-port 993) (nnimap-stream ssl)) nil)
  gnus-get-unread-articles(7 nil nil)
  gnus-group-get-new-news(nil)
  funcall-interactively(gnus-group-get-new-news nil)
  call-interactively(gnus-group-get-new-news nil nil)
  command-execute(gnus-group-get-new-news)

Admittedly, I am now using GMail directly to store marks whereas all my
other mail *was* going through my local dovecot instance (which I sync
with GMail and another server in the background), but it still seems
like it should be more reliable than this.

The reason I decided to go straight to GMail was that I wasn't sure how
Gnus-Cloud would behave if I happened to put my laptop to sleep before
its changes got sync'd out to the server and started reading mail
elsewhere.  Is there any provision for resolving changes from multiple
sources?

Thannks,

-- 
-Dave