Emacs Cloud

Emacs Cloud

[Lars Ingebrigtsen]

I think I'm now current on all the outstanding patches, so tomorrow I'm
gonna start on the Emacs Cloud stuff previously discussed here.

I wrote it up slightly on dar blogz:

http://lars.ingebrigtsen.no/2014/02/01/emacs-cloud/

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Ted Zlatanov]

On Fri, 31 Jan 2014 20:55:02 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> I think I'm now current on all the outstanding patches, so tomorrow I'm
LI> gonna start on the Emacs Cloud stuff previously discussed here.

LI> I wrote it up slightly on dar blogz:

LI> http://lars.ingebrigtsen.no/2014/02/01/emacs-cloud/

*cheering*

Regarding the bootstrap question, I think it's acceptable to require one
authenticated IMAP connection (from auth-source) to bootstrap the entire
Emacs setup.

I also want to request that you make the chunk protocol usable from
outside Gnus, so BBDB and other tools can use it too.

I am very excited about this.

Ted

Re: Emacs Cloud

[Rasmus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 31 Jan 2014 20:55:02 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 
>
> LI> I think I'm now current on all the outstanding patches, so tomorrow I'm
> LI> gonna start on the Emacs Cloud stuff previously discussed here.
>
> LI> I wrote it up slightly on dar blogz:
>
> LI> http://lars.ingebrigtsen.no/2014/02/01/emacs-cloud/

Sounds great!  I've been trying to keep my Gnus' (plural) updated
across machines with git-annex, but this would (presumably) be so much
better this way.

I vote for being able to store all info on /one/ IMAP-server even when
having several IMAP connections.  My personal IMAP is very fast and my
outlook.com-account is very slow.

> I also want to request that you make the chunk protocol usable from
> outside Gnus, so BBDB and other tools can use it too.

That would indeed be cool.  Gnus harvest is what I mainly use for
emails these days.

-- 
A page of history is worth a volume of logic

Re: Emacs Cloud

[Steinar Bang]

>>>>> Rasmus <rasmus@gmx.us>:

> I vote for being able to store all info on /one/ IMAP-server even when
> having several IMAP connections.  My personal IMAP is very fast and my
> outlook.com-account is very slow.

I use Ted's gnus-sync.el v2 with CouchDB backing, and have done so for
two years(?) now.

Re: Emacs Cloud

[Rasmus]

Hi Steinar,

Steinar Bang <sb@dod.no> writes:

>> I vote for being able to store all info on /one/ IMAP-server even when
>> having several IMAP connections.  My personal IMAP is very fast and my
>> outlook.com-account is very slow.
>
> I use Ted's gnus-sync.el v2 with CouchDB backing, and have done so for
> two years(?) now.

Thanks for pointing me to gnus-sync.  How did I miss it before?  I
have a question, though; I'd like to sync subscriptions as well as
read status.  I have this in my init file (I don't have a separate
dot-gnus).

   (setq gnus-sync-backend "/ssh:loging@server:~/gs.gpg"
         gnus-sync-file-encrypt-to "ABCD"
         gnus-sync-global-vars '(gnus-newsrc-last-checked-date)
         gnus-sync-newsrc-groups '("nntp" "nnrss")
         gnus-sync-newsrc-offsets '(2 3))

Stuff is synced whenever you'd expect it to be synced.  But my
gnus-group buffer still shows different groups/subscriptions.  I
looked for variables I could add to gs-global-vars that might serve,
but I didn't find any.

I'm probably missing something obvious.

Thanks,
Rasmus

-- 
If you can mix business and politics wonderful things can happen!

Re: Emacs Cloud

[Ted Zlatanov]

On Sat, 01 Feb 2014 21:23:47 +0100 Rasmus <rasmus@gmx.us> wrote: 

R> Hi Steinar,
R> Steinar Bang <sb@dod.no> writes:

>>> I vote for being able to store all info on /one/ IMAP-server even when
>>> having several IMAP connections.  My personal IMAP is very fast and my
>>> outlook.com-account is very slow.
>> 
>> I use Ted's gnus-sync.el v2 with CouchDB backing, and have done so for
>> two years(?) now.

R> Thanks for pointing me to gnus-sync.  How did I miss it before?  I
R> have a question, though; I'd like to sync subscriptions as well as
R> read status.  I have this in my init file (I don't have a separate
R> dot-gnus).

R>    (setq gnus-sync-backend "/ssh:loging@server:~/gs.gpg"
R>          gnus-sync-file-encrypt-to "ABCD"
R>          gnus-sync-global-vars '(gnus-newsrc-last-checked-date)
R>          gnus-sync-newsrc-groups '("nntp" "nnrss")
R>          gnus-sync-newsrc-offsets '(2 3))

R> Stuff is synced whenever you'd expect it to be synced.  But my
R> gnus-group buffer still shows different groups/subscriptions.  I
R> looked for variables I could add to gs-global-vars that might serve,
R> but I didn't find any.

R> I'm probably missing something obvious.

Don't use gnus-sync.  It doesn't work well (see past discussions here).
Lars' work on gnus-cloud will deprecate it.

Ted

Re: Emacs Cloud

[Andreas Schwab]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Don't use gnus-sync.  It doesn't work well (see past discussions here).
> Lars' work on gnus-cloud will deprecate it.

Definitely not.  gnus-sync has its use beyond gnus-cloud.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs Cloud

[Ted Zlatanov]

On Sat, 01 Feb 2014 22:50:58 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Don't use gnus-sync.  It doesn't work well (see past discussions here).
>> Lars' work on gnus-cloud will deprecate it.

AS> Definitely not.  gnus-sync has its use beyond gnus-cloud.

Can you elaborate?

Ted

Re: Emacs Cloud

[Andreas Schwab]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sat, 01 Feb 2014 22:50:58 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>
> AS> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> Don't use gnus-sync.  It doesn't work well (see past discussions here).
>>> Lars' work on gnus-cloud will deprecate it.
>
> AS> Definitely not.  gnus-sync has its use beyond gnus-cloud.
>
> Can you elaborate?

Its simplicity rules.  A plain file that can be stored anywhere.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs Cloud

[Ted Zlatanov]

On Sun, 02 Feb 2014 09:23:05 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Sat, 01 Feb 2014 22:50:58 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>> 
AS> Ted Zlatanov <tzz@lifelogs.com> writes:
>>>> Don't use gnus-sync.  It doesn't work well (see past discussions here).
>>>> Lars' work on gnus-cloud will deprecate it.
>> 
AS> Definitely not.  gnus-sync has its use beyond gnus-cloud.
>> 
>> Can you elaborate?

AS> Its simplicity rules.  A plain file that can be stored anywhere.

I'm glad you like it.  I'll probably rewrite it, then, if there are
fans, to either use a file or the gnus-cloud backend.  The CouchDB
backend is going away, I hope no one is sad about that.

Ted

Re: Emacs Cloud

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Don't use gnus-sync.  It doesn't work well (see past discussions here).

Indeed.

> Lars' work on gnus-cloud will deprecate it.

And I can't wait to start using the new gnus-cloud stuff. :-)

Re: Emacs Cloud

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Regarding the bootstrap question, I think it's acceptable to require one
> authenticated IMAP connection (from auth-source) to bootstrap the entire
> Emacs setup.

Since we're storing .authinfo on the IMAP server, too, I think we have
to require one additional encryption password here (in addition to the
IMAP credentials)...

> I also want to request that you make the chunk protocol usable from
> outside Gnus, so BBDB and other tools can use it too.

Sure.  It should be pretty general.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Ted Zlatanov]

On Sat, 01 Feb 2014 12:48:55 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Regarding the bootstrap question, I think it's acceptable to require one
>> authenticated IMAP connection (from auth-source) to bootstrap the entire
>> Emacs setup.

LI> Since we're storing .authinfo on the IMAP server, too, I think we have
LI> to require one additional encryption password here (in addition to the
LI> IMAP credentials)...

I hope that's optional and simply an auth-source backend (let me know if
you need help implementing that).  I'd rather not put my authinfo on an
IMAP server, even if encrypted:

- mail folders get wiped by accident and that info is too important

- the mail admin has full access to the spool

- having no IMAP connectivity means you have no passwords for other
  services

- it's not in a file, so bugs in EPA/EPG or elsewhere could result in
  storing unencrypted data on the IMAP server

- can't version it with Git, like I do now

Others might feel it's OK, of course.

Ted

Re: Emacs Cloud

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I hope that's optional and simply an auth-source backend (let me know if
> you need help implementing that).  I'd rather not put my authinfo on an
> IMAP server, even if encrypted:
>
> - mail folders get wiped by accident and that info is too important

It's still stored locally -- we're syncing local files via IMAP, not
just storing them there...

> - the mail admin has full access to the spool

But if they're encrypted, it lessens the problems.  But, yes, I can see
why (some) people wouldn't want to store them on the IMAP server...

And you may still use a .authinfo.gpg file, so it's doubly encrypted.
Twice as safe!!!1!1!ONE

> - having no IMAP connectivity means you have no passwords for other
>   services
>
> - it's not in a file, so bugs in EPA/EPG or elsewhere could result in
>   storing unencrypted data on the IMAP server
>
> - can't version it with Git, like I do now

See 1).  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Lars Ingebrigtsen]

Uhm...  how does one do symmetric encryption in Emacs, anyway?  There
seems to be a billion functions, but I just need one that does "prompt
for a password (via auth-source) and encrypt the contents in the
buffer"...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Ted Zlatanov]

On Sat, 01 Feb 2014 14:32:54 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Uhm...  how does one do symmetric encryption in Emacs, anyway?  There
LI> seems to be a billion functions, but I just need one that does "prompt
LI> for a password (via auth-source) and encrypt the contents in the
LI> buffer"...

Look at `auth-source-epa-make-gpg-token' and
`auth-source-epa-extract-gpg-token'.

Ted

Re: Emacs Cloud

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Look at `auth-source-epa-make-gpg-token' and
> `auth-source-epa-extract-gpg-token'.

Hm.  What's the role of the "secret" in those functions?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> Look at `auth-source-epa-make-gpg-token' and
>> `auth-source-epa-extract-gpg-token'.
>
> Hm.  What's the role of the "secret" in those functions?

And the cloud password should probably be saved in .authinfo, so it
should be fed into the epg functions somehow.  But it seems to want to
prompt for the password...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Ted Zlatanov]

On Sat, 01 Feb 2014 21:21:06 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Lars Ingebrigtsen <larsi@gnus.org> writes:
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>> 
>>> Look at `auth-source-epa-make-gpg-token' and
>>> `auth-source-epa-extract-gpg-token'.
>> 
>> Hm.  What's the role of the "secret" in those functions?

LI> And the cloud password should probably be saved in .authinfo, so it
LI> should be fed into the epg functions somehow.  But it seems to want to
LI> prompt for the password...

GnuPG 1.x can be automated so it doesn't ask for the passphrase (it can
pass through Emacs).  With 2.x that's intentionally disabled and you
have to use the daemon, which makes it impossible to supply the
passphrase externally.  See the docs for
`epa-file-cache-passphrase-for-symmetric-encryption' which Ueno-san
updated recently.

See
http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00156.html
and http://comments.gmane.org/gmane.emacs.devel/163708 for more
background.

I wrote a large patch to provide ciphers and encryption in Emacs
natively, hoping to avoid the GnuPG issues, but Stefan rejected it
because he wants to move in the FFI direction, so he wants to avoid
tight C bindings to GnuTLS or libnettle/libhogweed.  I plan to work on
this after the freeze so assume there will be something in the Emacs
trunk in 2014 that will do native encryption without prompting you for a
passphrase.

Ted

Re: Emacs Cloud

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> GnuPG 1.x can be automated so it doesn't ask for the passphrase (it can
> pass through Emacs).  With 2.x that's intentionally disabled and you
> have to use the daemon, which makes it impossible to supply the
> passphrase externally.  See the docs for
> `epa-file-cache-passphrase-for-symmetric-encryption' which Ueno-san
> updated recently.

Geez.  That's really unfortunate.  Having Gnus Cloud ask for the
password every single time makes this a no go.

Is there no other way to do symmetric encryption from Emacs?

> I wrote a large patch to provide ciphers and encryption in Emacs
> natively, hoping to avoid the GnuPG issues, but Stefan rejected it
> because he wants to move in the FFI direction, so he wants to avoid
> tight C bindings to GnuTLS or libnettle/libhogweed.  I plan to work on
> this after the freeze so assume there will be something in the Emacs
> trunk in 2014 that will do native encryption without prompting you for a
> passphrase.

Perhaps we can convince Stefan to reconsider.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

On Sun, Feb 02 2014, Lars Ingebrigtsen wrote:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> GnuPG 1.x can be automated so it doesn't ask for the passphrase (it can
>> pass through Emacs).  With 2.x that's intentionally disabled and you
>> have to use the daemon, which makes it impossible to supply the
>> passphrase externally.  See the docs for
>> `epa-file-cache-passphrase-for-symmetric-encryption' which Ueno-san
>> updated recently.
>
> Geez.  That's really unfortunate.  Having Gnus Cloud ask for the
> password every single time makes this a no go.
>
> Is there no other way to do symmetric encryption from Emacs?

What about using asymmetric?

-- 
Julien Danjou
# Free Software hacker # independent consultant
# http://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Emacs Cloud

[Lars Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> What about using asymmetric?

Perhaps?  How would that work in the cloud case?

(As you may have guessed, I know very little about encryption.  >"?)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On Mon, Feb 03 2014, Lars Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
>
>> What about using asymmetric?
>
> Perhaps?  How would that work in the cloud case?
>
> (As you may have guessed, I know very little about encryption.  >"?)

What I've in mind is that Gnus could use my public GPG key to encrypt
the file before uploading it. Then it would need my private key file to
decrypt it – and since I'm using a GPG agent, I've no need to enter any
passphrase, so Gnus can decrypt the file right away.

So:

Pros: no passphrase to enter, GPG agent takes care of it

Cons: need to have your secret key around – which might or might be not
      a problem depending where you run Gnus

-- 
Julien Danjou
# Free Software hacker # independent consultant
# http://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Emacs Cloud

[Lars Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Pros: no passphrase to enter, GPG agent takes care of it

I don't think we can rely on GPG agent being available...

> Cons: need to have your secret key around – which might or might be not
>       a problem depending where you run Gnus

If we have to sync over the private key to all places where we're
running Gnus, then that lessens the fun of just being able to say
`M-x gnus'.

Hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[David Engster]

Lars Ingebrigtsen writes:
> Julien Danjou <julien@danjou.info> writes:
>
>> Pros: no passphrase to enter, GPG agent takes care of it
>
> I don't think we can rely on GPG agent being available...

GnuPG 2.x *requires* the agent; it does not make sense to use GnuPG 2.x
without setting up the agent as well. It was a deliberate design
decision to have *one* secure daemon for querying and caching
pass-phrases, instead of leaving this to the applications, which will
often do it wrong.

If you really don't want to use the agent, then just keep using GnuPG
1.x, which is actively maintained (and probably will be for a long
time). Note that you can install GnuPG 1.x alongside GnuPG 2.x with no
problems, and every recent distribution I know has packages for both.

-David

Re: Emacs Cloud

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

> Note that you can install GnuPG 1.x alongside GnuPG 2.x with no
> problems, and every recent distribution I know has packages for both.

As far as I can see, Arch has only the current 2.0.22 version in its
official repository.  There's a gnupg1 package with version 1.4.16 in
the user repository, though, but its unlikely that people have that
installed.

Bye,
Tassilo

Re: Emacs Cloud

[David Engster]

Tassilo Horn writes:
> David Engster <deng@randomsample.de> writes:
>
>> Note that you can install GnuPG 1.x alongside GnuPG 2.x with no
>> problems, and every recent distribution I know has packages for both.
>
> As far as I can see, Arch has only the current 2.0.22 version in its
> official repository.  There's a gnupg1 package with version 1.4.16 in
> the user repository, though, but its unlikely that people have that
> installed.

Oh well, Arch users run into problems all the time anyway. They are
usually also not afraid of doing some reading, so they'll have to learn
about the agent and set it up properly.

-David

Re: Emacs Cloud

[Ted Zlatanov]

On Mon, 03 Feb 2014 15:58:24 +0100 David Engster <deng@randomsample.de> wrote: 

DE> Tassilo Horn writes:
>> David Engster <deng@randomsample.de> writes:
>> 
>>> Note that you can install GnuPG 1.x alongside GnuPG 2.x with no
>>> problems, and every recent distribution I know has packages for both.
>> 
>> As far as I can see, Arch has only the current 2.0.22 version in its
>> official repository.  There's a gnupg1 package with version 1.4.16 in
>> the user repository, though, but its unlikely that people have that
>> installed.

DE> Oh well, Arch users run into problems all the time anyway. They are
DE> usually also not afraid of doing some reading, so they'll have to learn
DE> about the agent and set it up properly.

It's not always an option.  Please don't explain this away as a user
problem that can be remedied by education and effort.  Not everyone
matches the "single user, single desktop" use case.

Ted

Re: Emacs Cloud

[David Engster]

Ted Zlatanov writes:
> On Mon, 03 Feb 2014 15:58:24 +0100 David Engster <deng@randomsample.de> wrote: 
>
> DE> Tassilo Horn writes:
>>> David Engster <deng@randomsample.de> writes:
>>> 
>>>> Note that you can install GnuPG 1.x alongside GnuPG 2.x with no
>>>> problems, and every recent distribution I know has packages for both.
>>> 
>>> As far as I can see, Arch has only the current 2.0.22 version in its
>>> official repository.  There's a gnupg1 package with version 1.4.16 in
>>> the user repository, though, but its unlikely that people have that
>>> installed.
>
> DE> Oh well, Arch users run into problems all the time anyway. They are
> DE> usually also not afraid of doing some reading, so they'll have to learn
> DE> about the agent and set it up properly.
>
> It's not always an option.  Please don't explain this away as a user
> problem that can be remedied by education and effort.  Not everyone
> matches the "single user, single desktop" use case.

I don't agree, but let's leave it at that. I've used the agent for years
under various setups with no problems (and yes, that includes remote
headless servers and the like). Anyone interested can read bug #15553
and gmane.emacs.devel/163708 and make up his own mind.

-David

Re: Emacs Cloud (coverage and killed groups)

[Lars Ingebrigtsen]

The file duplication thingies are basically done now, so I'm starting in
on the newsrc replication.

(Things are taking a lot longer than I expected because of Stuff.)

I had originally planned to just say "all network based servers should
be covered by default", but I think that's not really the common use
case.  I have a gazillion servers in my .newsrc.eld at home, but I
really just want to have news.gmane.org, news.gnus.org and one of my
IMAP servers while travelling.

So I think a new server mode command would be the thing -- to mark
server explicitly for coverage.

The other thing I'm wondering about what to do with is
`gnus-killed-list' and killed groups in general.

First of all, the list tends to be long:

(length gnus-killed-list)
=> 9138

So it would be nice not to carry that around all the time.  But if we
don't, then backends that don't support "give me new groups since DATE"
will pop up new groups that we have already killed off in one of the
other instances.

The other question is whether to replicate killing groups at all.  If
I'm on holiday, and I kill off 30 work-related groups, do I really want
the same to happen at home?  I think the answer is...  perhaps...

Any thoughts?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud (coverage and killed groups)

[Lars Ingebrigtsen]

I had originally thought of the Cloud stuff being totally symmetrical,
but perhaps that's not the best approach here.

That is, at home I have a Gnus setup that contains a gazillion servers,
and some of them are local, like nnml ones.  These obviously shouldn't
be replicated to other instances.  Fine.

But what happens to the topic topology?  If we push out newsrc data that
contains only these replicated groups initially (and I think that's the
right solution), what happens if we move a group from one topic to
another in one of the other instances?  We can't just do a full new dump
of the topology back to the "home" instance, because it no longer
contains all the groups...

So I'm now thinking we have to have a strict division between one "home"
instance that contains everything, and that can do full chunk dumps.
And then all the other instances would just log updates like

"read articles 56-69 in group foo.bar"
"moved group foo.bar from topic Zot to topic Bar"

The other non-home instances would pull these in as needed, and the
"home" instance, too.

The main problem, though, is that if you're not on your home instance
for months, the number of incremental chunks might get pretty ... huge.

But that may not actually be much of a problem.

Hm...

What do all y'all think?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud (coverage and killed groups)

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1040 bytes --]

On Fri, Feb 07 2014, Lars Ingebrigtsen wrote:

> What do all y'all think?

That might somewhat orthogonal – but at least it's once of my wish: I
would really like to have things like topics, group level, etc, be
stored in Gnus configuration rather than .newsrc.

For the kind of setup you describe (multiple and different servers),
that would be much helpful as the user would be able to program how he
wants to organize topics.
Same goes for group level; each time a new group appears in my IMAP
server, it has a group level of 3 whereas if it matches some strings,
it's going to be moved to 1 or 2 for example.

Having the ability to have Gnus configuration be the reference for this
information rather than .newsrc would be a large improvement IMHO.

The only thing that Gnus cloud should sync IMHO is the read status of
non-IMAP groups; for IMAP I wish Gnus would stop generating .newsrc
altogether.

My 2¢,

-- 
Julien Danjou
;; Free Software hacker ; independent consultant
;; http://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Emacs Cloud (coverage and killed groups)

[Peter Münster]

On Fri, Feb 07 2014, Lars Ingebrigtsen wrote:

> I had originally thought of the Cloud stuff being totally symmetrical,
> but perhaps that's not the best approach here.
>
> That is, at home I have a Gnus setup that contains a gazillion servers,
> and some of them are local, like nnml ones.  These obviously shouldn't
> be replicated to other instances.  Fine.

Why not? You could replicate everything, also the local groups. Just the
messages won't be available on every instance.

-- 
           Peter

Re: Emacs Cloud (coverage and killed groups)

[Lars Ingebrigtsen]

Peter Münster <pmlists@free.fr> writes:

> On Fri, Feb 07 2014, Lars Ingebrigtsen wrote:
>
>> I had originally thought of the Cloud stuff being totally symmetrical,
>> but perhaps that's not the best approach here.
>>
>> That is, at home I have a Gnus setup that contains a gazillion servers,
>> and some of them are local, like nnml ones.  These obviously shouldn't
>> be replicated to other instances.  Fine.
>
> Why not? You could replicate everything, also the local groups. Just the
> messages won't be available on every instance.

Displaying a gazillion dead and unusable groups doesn't seem like a good idea.

Besides, if you have an nnml group on your non-home setup, it's apt to
start pulling down data via POP, which is definitely not what you want.
Some messages would be accessible only on your laptop, while the rest
are at home.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud (coverage and killed groups)

[Ted Zlatanov]

On Thu, 06 Feb 2014 18:49:09 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> I had originally thought of the Cloud stuff being totally symmetrical,
LI> but perhaps that's not the best approach here.

LI> That is, at home I have a Gnus setup that contains a gazillion servers,
LI> and some of them are local, like nnml ones.  These obviously shouldn't
LI> be replicated to other instances.  Fine.

LI> But what happens to the topic topology?  If we push out newsrc data that
LI> contains only these replicated groups initially (and I think that's the
LI> right solution), what happens if we move a group from one topic to
LI> another in one of the other instances?  We can't just do a full new dump
LI> of the topology back to the "home" instance, because it no longer
LI> contains all the groups...

I would have a special "cloud" topic at the top level.  Or a "cloudify"
per-group property (not per server!)

Groups that have "cloudify" are then synchronized, together with their
topology position.  So the group would know it's under topic G, and if G
doesn't exist, it will be created.  Changing the topology of a group is
a configuration change like any other.  Again, this is *per group* so
you don't have a global topology map with cloudified groups.

I think inverting the gnus-cloud configuration from
global-descending-to-local to per-group is key here, but it does
complicate the interaction with the Gnus state.

LI> So I'm now thinking we have to have a strict division between one "home"
LI> instance that contains everything, and that can do full chunk dumps.
LI> And then all the other instances would just log updates

That's good.  But it should simply be a choice of "this instance
subscribes to cloudified groups X, Y, and 'gmane.*'" instead of making
such special cases.

LI> The main problem, though, is that if you're not on your home instance
LI> for months, the number of incremental chunks might get pretty ... huge.

LI> But that may not actually be much of a problem.

Nowadays, probably not.  Fast connections are pretty common.  Even a 3G
connection is quite fast (the lowest network speed I would expect from
our user base).

Ted

Re: Emacs Cloud

[Ted Zlatanov]

On Mon, 03 Feb 2014 09:08:47 +0100 David Engster <deng@randomsample.de> wrote: 

DE> Lars Ingebrigtsen writes:
>> Julien Danjou <julien@danjou.info> writes:
>> 
>>> Pros: no passphrase to enter, GPG agent takes care of it
>> 
>> I don't think we can rely on GPG agent being available...

DE> GnuPG 2.x *requires* the agent; it does not make sense to use GnuPG 2.x
DE> without setting up the agent as well. It was a deliberate design
DE> decision to have *one* secure daemon for querying and caching
DE> pass-phrases, instead of leaving this to the applications, which will
DE> often do it wrong.

I'd argue that Emacs is not an application in any way.

DE> If you really don't want to use the agent, then just keep using GnuPG
DE> 1.x, which is actively maintained (and probably will be for a long
DE> time). Note that you can install GnuPG 1.x alongside GnuPG 2.x with no
DE> problems, and every recent distribution I know has packages for both.

I've covered all of this with Stefan and Daiki Ueno in the discussions I
referenced, from my reasoning to annoying issues with the GPG daemon's pinentry.

Ted

Re: Emacs Cloud

[David Engster]

Ted Zlatanov writes:
> On Mon, 03 Feb 2014 09:08:47 +0100 David Engster <deng@randomsample.de> wrote: 
>
> DE> Lars Ingebrigtsen writes:
>>> Julien Danjou <julien@danjou.info> writes:
>>> 
>>>> Pros: no passphrase to enter, GPG agent takes care of it
>>> 
>>> I don't think we can rely on GPG agent being available...
>
> DE> GnuPG 2.x *requires* the agent; it does not make sense to use GnuPG 2.x
> DE> without setting up the agent as well. It was a deliberate design
> DE> decision to have *one* secure daemon for querying and caching
> DE> pass-phrases, instead of leaving this to the applications, which will
> DE> often do it wrong.
>
> I'd argue that Emacs is not an application in any way.

Sorry, I don't get it.

> DE> If you really don't want to use the agent, then just keep using GnuPG
> DE> 1.x, which is actively maintained (and probably will be for a long
> DE> time). Note that you can install GnuPG 1.x alongside GnuPG 2.x with no
> DE> problems, and every recent distribution I know has packages for both.
>
> I've covered all of this with Stefan and Daiki Ueno in the discussions I
> referenced, from my reasoning to annoying issues with the GPG daemon's pinentry.

I know, and I don't plan to repeat this discussion here. Using pinentry
has its problems, especially on remote terminals, but IMHO it makes no
sense to work against the design of GnuPG v2. If you don't like it, use
v1 instead (that's one reason why it's still maintained).

-David

Re: Emacs Cloud

[Ted Zlatanov]

On Sun, 02 Feb 2014 14:53:03 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> GnuPG 1.x can be automated so it doesn't ask for the passphrase (it can
>> pass through Emacs).  With 2.x that's intentionally disabled and you
>> have to use the daemon, which makes it impossible to supply the
>> passphrase externally.  See the docs for
>> `epa-file-cache-passphrase-for-symmetric-encryption' which Ueno-san
>> updated recently.

LI> Geez.  That's really unfortunate.  Having Gnus Cloud ask for the
LI> password every single time makes this a no go.

LI> Is there no other way to do symmetric encryption from Emacs?

Not to my knowledge.

>> I wrote a large patch to provide ciphers and encryption in Emacs
>> natively, hoping to avoid the GnuPG issues, but Stefan rejected it
>> because he wants to move in the FFI direction, so he wants to avoid
>> tight C bindings to GnuTLS or libnettle/libhogweed.  I plan to work on
>> this after the freeze so assume there will be something in the Emacs
>> trunk in 2014 that will do native encryption without prompting you for a
>> passphrase.

LI> Perhaps we can convince Stefan to reconsider.  >"?

Nah, just wait.  Use a stub ROT13 function for now.

Ted

Re: Emacs Cloud

[Ted Zlatanov]

On Sat, 01 Feb 2014 21:14:28 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Look at `auth-source-epa-make-gpg-token' and
>> `auth-source-epa-extract-gpg-token'.

LI> Hm.  What's the role of the "secret" in those functions?

When auth-source sees a token in authinfo that looks like this:

gpg:LS0tLS1CRUdJTiBQR1AgTUVTU0FHRS0tLS0tClZlcnNpb246IEdudVBHIHYxLjQuMTEgKEdOVS9MaW51eCkKCmpBMEVBd01DT25qMjB1ak9rZnRneVI3K21iNm9aZWhuLzRad3cySkdlbnVaKzRpeEswWDY5di9icDI1U1dsQT0KPS9yc2wKLS0tLS1FTkQgUEdQIE1FU1NBR0UtLS0tLQo=

it will BASE64-decode the data after "gpg:" and then do a GnuPG decode
of the resulting data from a temporary file.  This allows you to have
such tokens in an unencrypted authinfo file, effectively hiding the
password but leaving the connection info readable. These functions go
from encoded token to decoded secret and back.

Ted

Re: Emacs Cloud

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LI> Hm.  What's the role of the "secret" in those functions?
>
> When auth-source sees a token in authinfo that looks like this:
>
> gpg:LS0tLS1CRUdJTiBQR1AgTUVTU0FHRS0tLS0tClZlcnNpb246IEdudVBHIHYxLjQuMTEgKEdOVS9MaW51eCkKCmpBMEVBd01DT25qMjB1ak9rZnRneVI3K21iNm9aZWhuLzRad3cySkdlbnVaKzRpeEswWDY5di9icDI1U1dsQT0KPS9yc2wKLS0tLS1FTkQgUEdQIE1FU1NBR0UtLS0tLQo=
>
> it will BASE64-decode the data after "gpg:" and then do a GnuPG decode
> of the resulting data from a temporary file.  This allows you to have
> such tokens in an unencrypted authinfo file, effectively hiding the
> password but leaving the connection info readable. These functions go
> from encoded token to decoded secret and back.

Aha!  I totally misread that function.  I thought "mysecret" was some
kind of salt and "~/.netrc" was the thing being encrypted:

(auth-source-epa-make-gpg-token "mysecret" "~/.netrc")

But now I understand -- "mysecret" is the thing being encrypted.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Ted Zlatanov]

On Sat, 01 Feb 2014 13:44:51 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I hope that's optional and simply an auth-source backend (let me know if
>> you need help implementing that).  I'd rather not put my authinfo on an
>> IMAP server, even if encrypted:
>> 
>> - mail folders get wiped by accident and that info is too important

LI> It's still stored locally -- we're syncing local files via IMAP, not
LI> just storing them there...

Oh, I see.  You're building a Dropbox-like service, backed by IMAP and
for arbitrary files, not just everything under a directory.

>> - the mail admin has full access to the spool

LI> But if they're encrypted, it lessens the problems.  But, yes, I can see
LI> why (some) people wouldn't want to store them on the IMAP server...

LI> And you may still use a .authinfo.gpg file, so it's doubly encrypted.
LI> Twice as safe!!!1!1!ONE

Yes, eventually it will be authinfo.gpg.pgp :)

Ted

Re: Emacs Cloud

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

> I think I'm now current on all the outstanding patches, so tomorrow I'm
> gonna start on the Emacs Cloud stuff previously discussed here.

> I wrote it up slightly on dar blogz:

> http://lars.ingebrigtsen.no/2014/02/01/emacs-cloud/

Will old entries in the IMAP folder used for storage be expired? Or will
the imap folder used for storage continue to grow in size?

Re: Emacs Cloud

[Lars Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Will old entries in the IMAP folder used for storage be expired? Or will
> the imap folder used for storage continue to grow in size?

Old entries are deleted.  There will be only one "full" dump and then a
number of partial chunks.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Lars Ingebrigtsen]

Hm...  encryption in Emacs Lisp?  Can this be used, I wonder?

http://www.emacswiki.org/emacs-en/rijndael.el

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Ted Zlatanov]

On Wed, 05 Feb 2014 15:06:18 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Hm...  encryption in Emacs Lisp?  Can this be used, I wonder?
LI> http://www.emacswiki.org/emacs-en/rijndael.el

Doing Rijndael/AES in Emacs Lisp will be pretty slow as the gnus-cloud
data grows.

Daiki Ueno's patch (on emacs-devel) looks promising and I hope we can
continue with it.  I asked.  The FFI integration, regardless of the
libnettle/libhogweed discussion, will bring us the crypto primitives
soon.

So I would stick with the Base64 stub for now; I would only use the
ELisp solution as a last resort.

Ted

Re: Emacs Cloud

[Lars Ingebrigtsen]

Bleaurgh.  I think I've got a cold, so I'm not making much progress
here.  Instead I'm just fixing doc bugs tonight...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Daiki Ueno]

I wasn't really following the discussion, but I now suspect the use of
symmetric encryption here might be irrelevant in the first place.  Do
you plan to use untrusted (even authenticated, e.g. Gmail) IMAP servers
for file sharing, right?

If so, those symmetrically encrypted data can be a target of dictionary
attacks.  You will be giving unlimited time to attackers (or server
admins) cracking your encrypted data.  That's why people normally don't
want to upload their secret keys in ~/.ssh or ~/.gnupg (even if they are
password-protected by default).

Regards,
-- 
Daiki Ueno

Re: Emacs Cloud

[Ted Zlatanov]

On Mon, 10 Feb 2014 17:43:11 +0900 Daiki Ueno <ueno@gnu.org> wrote: 

DU> I wasn't really following the discussion, but I now suspect the use of
DU> symmetric encryption here might be irrelevant in the first place.  Do
DU> you plan to use untrusted (even authenticated, e.g. Gmail) IMAP servers
DU> for file sharing, right?

DU> If so, those symmetrically encrypted data can be a target of dictionary
DU> attacks.  You will be giving unlimited time to attackers (or server
DU> admins) cracking your encrypted data.

Primarily, the data is the Gnus configuration (including topology) and
group marks, the newsrc.eld IIUC.  Encrypting it may not justify the
full asymmetric infrastructure, especially with the use cases of "you're
a guest on a machine without the ability to install software" and
"you're a guest on a machine and you can't obtain a GnuPG agent
connection to a secure machine" (both common in corporate settings).

It's still a concern, though, as the Gnus newsrc can at least indicate
your reading habits.  I personally don't bother to hide or encrypt my
newsrc.eld on shared machines, but wouldn't want it easily exposed to
IMAP server admins.  I think it should be a user choice how that data
should be encrypted.

Lars mentioned he would like to put the authinfo data in the cloud as
well.  I think *that* should definitely require an explicit choice by
the user and it would make sense to apply asymmetric encryption to that
data specifically, if the authinfo is already encrypted (has the .gpg
extension).

Either way, I plan to treat the authinfo data as keychain entries rather
than one big block of data, storing them separately and allowing them to
be updated individually.  That would mitigate the dictionary attack risk
even with symmetric encryption.  auth-source.el will need some work, but
it's due for a refactoring anyhow.

The "encrypted tokens" I added and the plist support you added in
auth-source could also be useful here to mitigate the risk, since
decrypting the authinfo lines doesn't give access to the secrets.

However we approach it, I think the user should have some choice in the
matter and we should allow for the two use cases I mentioned above.

Ted

Re: Emacs Cloud

[Lars Ingebrigtsen]

Daiki Ueno <ueno@gnu.org> writes:

> I wasn't really following the discussion, but I now suspect the use of
> symmetric encryption here might be irrelevant in the first place.  Do
> you plan to use untrusted (even authenticated, e.g. Gmail) IMAP servers
> for file sharing, right?
>
> If so, those symmetrically encrypted data can be a target of dictionary
> attacks.  You will be giving unlimited time to attackers (or server
> admins) cracking your encrypted data.  That's why people normally don't
> want to upload their secret keys in ~/.ssh or ~/.gnupg (even if they are
> password-protected by default).

We're talking about storing the data at a host you trust enough that you
let them store your unencrypted mail, which probably reveals more about
you than any .newsrc data would ever do.

However, that doesn't mean that Gnus shouldn't try to make things safer
for you if you want.  Somebody that runs an IMAP server for you already
has some of your credentials on hand, as well as the data from a
thousand password resets emails.  Storing the .newsrc data with
symmetric encryption helps against 1) accidental data leakage (when
somebody is watching the output of tcpdump) or 2) idle curiosity (when
somebody has too much time on their hands and are grubbing through
data).  If it's a directed attack by the owners of the IMAP server,
you're probably screwed, anyway.

In addition to storing the newsrc data, I also want to store the
relevant bits from your .authinfo.  That's the really ticklish bits.
If you're connecting to a couple of news servers that require NNTP
passwords, those would also be stored in the encrypted chunks.  However,
those aren't the most secret secrets in the universe, anyway.  Most NNTP
servers don't even support TLS, so you're sending those credentials
using plain text.

So to sum up:

1) Carthage should be destroyed.

2) Symmetric encryption is good enough for this use case.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Emacs Cloud

[Daiki Ueno]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Somebody that runs an IMAP server for you already has some of your
> credentials on hand, as well as the data from a thousand password
> resets emails.

They are usually one-time use.  I'm talking about off-line dictionary
attacks.  They could use a gazillion of computers (off-line) cracking
your encryption passwords to see your credentials inside ~/.authinfo,
including ones for other IMAP servers.

> 2) Symmetric encryption is good enough for this use case.

I still don't get why you concluded this is "good enough".  IMHO, the
use of symmetric encryption here is nothing but obfuscation.  Why not
compress+base64 is good enough?

Anyway, I'd suggest to warn users about

- what data will be stored on remote server
- how it will be protected

when setting up, at least.

Regards,
-- 
Daiki Ueno