how do I subscribe to a group as a string from code?

how do I subscribe to a group as a string from code?

[Ted Zlatanov]

I need this for my gnus-sync.el work.  I have save and read working, now
I want to be able to sync subscriptions.  As soon as that's working I'll
put it out.

Given a string: "nntp+news.gwene.org:gwene.org.slashdot" I want to
subscribe to that group.  The server is not necessarily defined, but
it's OK to fail in that case.

I looked at `gnus-browse-unsubscribe-group'.  It's very tightly tied to
the server browsing buffer.  I can dig into the backend functionality
more, but was hoping someone could save me a few hours of poking around.

Thanks
Ted

Re: how do I subscribe to a group as a string from code?

[Tassilo Horn]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I need this for my gnus-sync.el work.  I have save and read working,
> now I want to be able to sync subscriptions.  As soon as that's
> working I'll put it out.
>
> Given a string: "nntp+news.gwene.org:gwene.org.slashdot" I want to
> subscribe to that group.  The server is not necessarily defined, but
> it's OK to fail in that case.

I've just tried

  (gnus-subscribe-newsgroup "nntp+Gmane:gwene.org.slashdot")

and that did the trick.  Do I understand your question wrongly, or did
you miss the forest for the trees?

Bye,
Tassilo
-- 
Sent from my Emacs

Re: how do I subscribe to a group as a string from code?

[Ted Zlatanov]

On Thu, 20 Oct 2011 21:07:50 +0200 Tassilo Horn <tassilo@member.fsf.org> wrote: 

TH> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I need this for my gnus-sync.el work.  I have save and read working,
>> now I want to be able to sync subscriptions.  As soon as that's
>> working I'll put it out.
>> 
>> Given a string: "nntp+news.gwene.org:gwene.org.slashdot" I want to
>> subscribe to that group.  The server is not necessarily defined, but
>> it's OK to fail in that case.

TH> I've just tried

TH>   (gnus-subscribe-newsgroup "nntp+Gmane:gwene.org.slashdot")

TH> and that did the trick.  Do I understand your question wrongly, or did
TH> you miss the forest for the trees?

Whoa, how the .... does that work?  I don't have a "Gmane" server
defined in my backends, but it subscribed to it and I can look at that
group on Gmane.  This also works in my original case, where I have that
foreign server defined:

(gnus-subscribe-newsgroup "nntp+news.gwene.org:gwene.org.slashdot")

so I guess it solves my problem, but I'm really curious why the "Gmane"
server worked without a definition and I am not sure if I can rely on
this behavior for general sync.

Ted

Re: how do I subscribe to a group as a string from code?

[Tassilo Horn]

Ted Zlatanov <tzz@lifelogs.com> writes:

> TH> I've just tried
>
> TH>   (gnus-subscribe-newsgroup "nntp+Gmane:gwene.org.slashdot")
>
> TH> and that did the trick.  Do I understand your question wrongly, or did
> TH> you miss the forest for the trees?
>
> Whoa, how the .... does that work?  I don't have a "Gmane" server
> defined in my backends, but it subscribed to it and I can look at that
> group on Gmane.  This also works in my original case, where I have
> that foreign server defined:
>
> (gnus-subscribe-newsgroup "nntp+news.gwene.org:gwene.org.slashdot")
>
> so I guess it solves my problem, but I'm really curious why the
> "Gmane" server worked without a definition and I am not sure if I can
> rely on this behavior for general sync.

Oh, I chose Gmane, cause that's the name of the select method with
nntp-server news.gmane.org here.  Not sure why that also works for you
without such a select method.  I wouldn't assume that in the general
case.

However, I'm pretty sure that'll always work if you feed in a valid
select method name.  Can't you just iterate the select methods comparing
the server addresses for picking the right one?

Well, ok, it's possible to have several select methods for the same
server, say, different imap accounts.  In that case, you are probably
lost.

Bye,
Tassilo
-- 
Sent from my Emacs

Re: how do I subscribe to a group as a string from code?

[Ted Zlatanov]

On Thu, 20 Oct 2011 22:11:40 +0200 Tassilo Horn <tassilo@member.fsf.org> wrote: 

TH> Ted Zlatanov <tzz@lifelogs.com> writes:

>> Whoa, how the .... does that work?  I don't have a "Gmane" server
>> defined in my backends, but it subscribed to it and I can look at that
>> group on Gmane.  This also works in my original case, where I have
>> that foreign server defined:
>> 
>> (gnus-subscribe-newsgroup "nntp+news.gwene.org:gwene.org.slashdot")
>> 
>> so I guess it solves my problem, but I'm really curious why the
>> "Gmane" server worked without a definition and I am not sure if I can
>> rely on this behavior for general sync.

TH> Oh, I chose Gmane, cause that's the name of the select method with
TH> nntp-server news.gmane.org here.  Not sure why that also works for you
TH> without such a select method.  I wouldn't assume that in the general
TH> case.

Exactly.  It's weird--I'd rather see it fail!

TH> However, I'm pretty sure that'll always work if you feed in a valid
TH> select method name.  Can't you just iterate the select methods comparing
TH> the server addresses for picking the right one?

Yup.  I'll assume it will work, since I ask y/n on subscription anyhow.

TH> Well, ok, it's possible to have several select methods for the same
TH> server, say, different imap accounts.  In that case, you are probably
TH> lost.

It's OK, these are backends that can't store marks, usually nntp.  You
don't normally see multiple accounts with those so I think it's OK.

Thanks for your help!
Ted

Re: how do I subscribe to a group as a string from code?

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> I need this for my gnus-sync.el work.  I have save and read working,
> now I want to be able to sync subscriptions.  As soon as that's
> working I'll put it out.

Cool!  Are we past the emacs 24 feature freeze?

gnus-sync.el v2 (was: how do I subscribe to a group as a string from code?)

[Ted Zlatanov]

On Fri, 21 Oct 2011 16:44:43 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> I need this for my gnus-sync.el work.  I have save and read working,
>> now I want to be able to sync subscriptions.  As soon as that's
>> working I'll put it out.

SB> Cool!  Are we past the emacs 24 feature freeze?

I'm not sure if that applies to Gnus; I think if Yamaoka-san does not
synchronize gnus-sync.el with the Emacs trunk, that's sufficient to make
us compliant with the Emacs feature freeze.  Katsumi, can you do that?
I won't commit before you confirm.

Regarding gnus-sync.el, the good news is that it works great, the bad
news is that you need a working CouchDB server.  I tried a standalone
server but CouchDB did what I needed much better.  The old sync method,
via a file, still works for those who don't want the new functionality.

I sync subscriptions (including level), newsgroup topics, read ranges,
and marks.  It works pretty well for me after a weekend test.

The biggest piece missing is username+password validation and I'm
working on that.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Regarding gnus-sync.el, the good news is that it works great, the bad
> news is that you need a working CouchDB server. 

Not a problem.  But do you run nekkid CouchDB on the big bad Internet?
Or is it hidden behind an apache, or some other front end?

> I tried a standalone server but CouchDB did what I needed much better.
> The old sync method, via a file, still works for those who don't want
> the new functionality.

Oh, I'll go for the new functionality.

> I sync subscriptions (including level), newsgroup topics, read ranges,
> and marks.  It works pretty well for me after a weekend test.

Ok.  What about servers?

> The biggest piece missing is username+password validation and I'm
> working on that.

Um... username+password for CouchDB?  Or are you sync'ing
username+password for different servers?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Mon, 24 Oct 2011 22:48:20 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> Regarding gnus-sync.el, the good news is that it works great, the bad
>> news is that you need a working CouchDB server. 

SB> Not a problem.  But do you run nekkid CouchDB on the big bad Internet?
SB> Or is it hidden behind an apache, or some other front end?

There's built-in security in CouchDB.  I am OK with the defaults but
CouchDB authentication is pluggable and you can configure things any way
you like.

Whether that's OK for everyone I can't say; they can always run things
through a proxy or VPN or whatever makes sense.  For me it's good
enough.

>> I sync subscriptions (including level), newsgroup topics, read ranges,
>> and marks.  It works pretty well for me after a weekend test.

SB> Ok.  What about servers?

No, the servers are not synced.  I don't think it's possible because
they are in the user configuration, not in newsrc.eld necessarily.  I
could sync just the foreign servers but that's even more complicated
between machines with possibly identical server names.

>> The biggest piece missing is username+password validation and I'm
>> working on that.

SB> Um... username+password for CouchDB?  Or are you sync'ing
SB> username+password for different servers?

As described in
http://wiki.apache.org/couchdb/Security_Features_Overview, I'll set up
an admin just for the CouchDB database.

I'll host other users if they need it.  Their sync DB can coexist with
mine.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> There's built-in security in CouchDB.  I am OK with the defaults but
> CouchDB authentication is pluggable and you can configure things any
> way you like.

I wasn't just thinking of the authentication.  I was thinking against
robustness against attacks when exposed on the Internet.

> Whether that's OK for everyone I can't say; they can always run things
> through a proxy or VPN or whatever makes sense.  For me it's good
> enough.

I'll do a little bit of research on what's possible.

SB> Ok.  What about servers?

> No, the servers are not synced.  I don't think it's possible because
> they are in the user configuration, not in newsrc.eld necessarily.  I
> could sync just the foreign servers but that's even more complicated
> between machines with possibly identical server names.

Hm... ok... (not sure I understand why this is more complicated than the
other stuff.  Will look at it later).

>>> The biggest piece missing is username+password validation and I'm
>>> working on that.

SB> Um... username+password for CouchDB?  Or are you sync'ing
SB> username+password for different servers?

> As described in
> http://wiki.apache.org/couchdb/Security_Features_Overview, I'll set up
> an admin just for the CouchDB database.

> I'll host other users if they need it.  Their sync DB can coexist with
> mine.
 
Ah, ok

One thing that will good is nice behaviour when the server can't be
contacted, ie. not hanging for a long time waiting for the server, and
no marks lost when the server can be contacted at a later stage.

This would be important when coexisting with the agent.

Re: gnus-sync.el v2

[Ted Zlatanov]

On Tue, 25 Oct 2011 17:57:45 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> There's built-in security in CouchDB.  I am OK with the defaults but
>> CouchDB authentication is pluggable and you can configure things any
>> way you like.

SB> I wasn't just thinking of the authentication.  I was thinking against
SB> robustness against attacks when exposed on the Internet.

That's a moving target I can't chase.  There is plenty of software to
handle this transparently (CouchDB is just a web server, after all) and
it's really, REALLY hard to do it well.

>> No, the servers are not synced.  I don't think it's possible because
>> they are in the user configuration, not in newsrc.eld necessarily.  I
>> could sync just the foreign servers but that's even more complicated
>> between machines with possibly identical server names.

SB> Hm... ok... (not sure I understand why this is more complicated than the
SB> other stuff.  Will look at it later).

I am not looking to synchronize the Gnus configuration, only the
topics, subscriptions, and marks.  It's much riskier to sync
the configuration, both in terms of bugs and support, and in terms of
security.  So for me it's not a priority.

>>>> The biggest piece missing is username+password validation and I'm
>>>> working on that.

SB> One thing that will good is nice behaviour when the server can't be
SB> contacted, ie. not hanging for a long time waiting for the server, and
SB> no marks lost when the server can be contacted at a later stage.

SB> This would be important when coexisting with the agent.

Right now it hangs hard but marks are never lost.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> I am not looking to synchronize the Gnus configuration, only the
> topics, subscriptions, and marks.  It's much riskier to sync the
> configuration, both in terms of bugs and support, and in terms of
> security.  So for me it's not a priority.

Ok.

SB> One thing that will good is nice behaviour when the server can't be
SB> contacted, ie. not hanging for a long time waiting for the server, and
SB> no marks lost when the server can be contacted at a later stage.

SB> This would be important when coexisting with the agent.

> Right now it hangs hard but marks are never lost.

Right, they will be saved in .newsrc.eld, but won't necessarily be
synchronized...?

Out of curiosity: How do you decide which way the synchronization of
marks should go?  (ie. from gnus to the server, or update the gnus mark
based on what's on the server)

Do you use time-stamping?  If so: how fine-grained?

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
>>>>> Ted Zlatanov <tzz@lifelogs.com>:

SB> One thing that will good is nice behaviour when the server can't be
SB> contacted, ie. not hanging for a long time waiting for the server, and
SB> no marks lost when the server can be contacted at a later stage.

SB> This would be important when coexisting with the agent.

>> Right now it hangs hard but marks are never lost.

Er,... I read that wrong, so the rest of my questions don't really make
sense.

Above sentence now read and parsed (hangs trying to connect, but won't
lose marks).

Re: gnus-sync.el v2

[Ted Zlatanov]

On Tue, 25 Oct 2011 19:16:01 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> Out of curiosity: How do you decide which way the synchronization of
SB> marks should go?  (ie. from gnus to the server, or update the gnus mark
SB> based on what's on the server)

SB> Do you use time-stamping?  If so: how fine-grained?

I save the time stamp but it's not used.  The way CouchDB works is the
revision gets incremented (it looks like "23-md5hashhere") on save and
you get it back.  You have to provide the latest revision to overwrite a
document.  So whoever saves the latest revision knows how to overwrite
it; if you are behind then you can't save your entry over the latest.

You need to read the latest entry (using `gnus-sync-read') which
currently overrides whatever you've done with the server's state.  I may
make it merge later but it hasn't been an issue for me, I never use two
Gnus concurrently.  Merging is not too bad in any case, you just need to
do a union of all the ranges.

I have the user+password working so now I just need Yamaoka-san to
confirm he won't push gnus-sync.el to Emacs, so we don't break the
Emacs feature freeze.  I'll commit right after.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> I have the user+password working so now I just need Yamaoka-san to
> confirm he won't push gnus-sync.el to Emacs, so we don't break the
> Emacs feature freeze.  I'll commit right after.

I'm looking forward to trying it out.

Re: gnus-sync.el v2

[Katsumi Yamaoka]

Ted Zlatanov <tzz@lifelogs.com> wrote:
[...]
> I have the user+password working so now I just need Yamaoka-san to
> confirm he won't push gnus-sync.el to Emacs, so we don't break the
> Emacs feature freeze.  I'll commit right after.

Confirmed.  Please go ahead.  Sorry for my lazy response.

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Katsumi Yamaoka <yamaoka@jpl.org>:

> Ted Zlatanov <tzz@lifelogs.com> wrote:
> [...]
>> I have the user+password working so now I just need Yamaoka-san to
>> confirm he won't push gnus-sync.el to Emacs, so we don't break the
>> Emacs feature freeze.  I'll commit right after.

> Confirmed.  Please go ahead.  Sorry for my lazy response.

Ok.  Before I pull, is there something I should do to preserve my
current marks?  I obviously also need to set up a new CouchDB server.
Are there instructions somewhere?  Ie. a special database that needs to
be created?  What port should I use?

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> Ok.  Before I pull, is there something I should do to preserve my
> current marks? 

Note: I will do the obvious, such as copying both .newsrc.eld and the
current sync file on the server

What I was thinking of was: how do I proceed into the sync'able future,
preserving all present marks.  Ie. how do I bring the current marks with
me.

> I obviously also need to set up a new CouchDB server.  Are there
> instructions somewhere?  Ie. a special database that needs to be
> created?  What port should I use?

I will start with apt-get'ing CouchDB

I'm running debian stable, which has 0.11.0-2.3.  
 http://packages.debian.org/squeeze/couchdb

Is that new enough?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Thu, 27 Oct 2011 11:42:47 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Katsumi Yamaoka <yamaoka@jpl.org>:
>> Ted Zlatanov <tzz@lifelogs.com> wrote:
>> [...]
>>> I have the user+password working so now I just need Yamaoka-san to
>>> confirm he won't push gnus-sync.el to Emacs, so we don't break the
>>> Emacs feature freeze.  I'll commit right after.

>> Confirmed.  Please go ahead.  Sorry for my lazy response.

SB> Ok.  Before I pull, is there something I should do to preserve my
SB> current marks?  I obviously also need to set up a new CouchDB server.
SB> Are there instructions somewhere?  Ie. a special database that needs to
SB> be created?  What port should I use?

Set `gnus-sync-backend' to '(lesync "http://yoururl:yourport/yourdatabase")

Then set it up, gnus-sync.el has a function for that too.  I gave
examples in the commentary.  Read it carefully.  It should be really easy.

As long as you don't call `gnus-sync-read' nothing will happen to break
your setup.  So call `gnus-sync-save' and admire the results in the
database.  Make sure they look reasonable.  The admin interface is at
http://yoururl:yourport/yourdatabase/_utils/index.html

When you're ready, call `gnus-sync-save' then you must `g' to get
everything updated, same as the old gnus-sync.el.  You may want to set
`gnus-verbose' to 10 and watch *Messages* carefully.

It's all experimental so please keep good backups :)

On Thu, 27 Oct 2011 13:23:31 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> Note: I will do the obvious, such as copying both .newsrc.eld and the
SB> current sync file on the server

SB> What I was thinking of was: how do I proceed into the sync'able future,
SB> preserving all present marks.  Ie. how do I bring the current marks with
SB> me.

Just save from where you want to keep the marks.

>> I obviously also need to set up a new CouchDB server.  Are there
>> instructions somewhere?  Ie. a special database that needs to be
>> created?  What port should I use?

SB> I will start with apt-get'ing CouchDB

SB> I'm running debian stable, which has 0.11.0-2.3.  
SB>  http://packages.debian.org/squeeze/couchdb

SB> Is that new enough?

I'd go with the 1.x releases, if possible.  I have used 1.0.1 while
testing.  I don't know if that's close to 0.11 or not, sorry.

Ted

Re: gnus-sync.el v2

[Ted Zlatanov]

On Thu, 27 Oct 2011 13:36:01 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> When you're ready, call `gnus-sync-save' then you must `g' to get
TZ> everything updated, same as the old gnus-sync.el.  You may want to set
TZ> `gnus-verbose' to 10 and watch *Messages* carefully.

s/gnus-sync-save/gnus-sync-read/ of course.  Sorry.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

Another config item I'm confused about:
 ;; against a LeSync server (beware the vampire LeSync, who knows your newsrc)

 ;; (setq gnus-sync-backend '(lesync "http://lesync.info/sync.php")
 ;;       gnus-sync-global-vars `(gnus-newsrc-last-checked-date)
 ;;       gnus-sync-newsrc-groups `("nntp" "nnrss")
 ;;       gnus-sync-newsrc-offsets `(2 3))

Is the local part of that URL right?  Ie. "sync.php"?

Also: is there a way for me to transition my existing Tramp+EPA sync
file into the lesync couchdb database?  Since merging of read in the
.newsrc.eld in gnus-sync v1 didn't work, I will probably lose a lot of
marks in the transition otherwise...?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Thu, 03 Nov 2011 22:09:34 +0100 Steinar Bang <sb@dod.no> wrote: 

SB> Another config item I'm confused about:
SB>  ;; against a LeSync server (beware the vampire LeSync, who knows your newsrc)

SB>  ;; (setq gnus-sync-backend '(lesync "http://lesync.info/sync.php")
SB>  ;;       gnus-sync-global-vars `(gnus-newsrc-last-checked-date)
SB>  ;;       gnus-sync-newsrc-groups `("nntp" "nnrss")
SB>  ;;       gnus-sync-newsrc-offsets `(2 3))

SB> Is the local part of that URL right?  Ie. "sync.php"?

Sorry, I missed that in the instructions.  Fixed.

SB> Also: is there a way for me to transition my existing Tramp+EPA sync
SB> file into the lesync couchdb database?  Since merging of read in the
SB> .newsrc.eld in gnus-sync v1 didn't work, I will probably lose a lot of
SB> marks in the transition otherwise...?

Your existing newsrc will be saved as is to LeSync.  `gnus-sync-save'
is 100% safe, it will never ever change your local marks, and it won't
write if the revision has changed (meaning someone else has updated, in
which case you need to `gnus-sync-read' first).  So save and then use
Futon (http://lesync.info:5984/_utils for example) to look at the data
to be sure it's OK.

`gnus-sync-read' will, currently, overwrite all local marks with what
LeSync says, and update the revision for each newsgroup.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Your existing newsrc will be saved as is to LeSync.  `gnus-sync-save'
> is 100% safe, it will never ever change your local marks, and it won't
> write if the revision has changed (meaning someone else has updated, in
> which case you need to `gnus-sync-read' first).  So save and then use
> Futon (http://lesync.info:5984/_utils for example) to look at the data
> to be sure it's OK.

> `gnus-sync-read' will, currently, overwrite all local marks with what
> LeSync says, and update the revision for each newsgroup.

Um... could I gnus-sync-read the old Tramp+EPA file, and then
gnus-sync-write the combined results into the lesync couchdb?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Thu, 03 Nov 2011 23:37:36 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> Your existing newsrc will be saved as is to LeSync.  `gnus-sync-save'
>> is 100% safe, it will never ever change your local marks, and it won't
>> write if the revision has changed (meaning someone else has updated, in
>> which case you need to `gnus-sync-read' first).  So save and then use
>> Futon (http://lesync.info:5984/_utils for example) to look at the data
>> to be sure it's OK.

>> `gnus-sync-read' will, currently, overwrite all local marks with what
>> LeSync says, and update the revision for each newsgroup.

SB> Um... could I gnus-sync-read the old Tramp+EPA file, and then
SB> gnus-sync-write the combined results into the lesync couchdb?

1) set your gnus-sync-backend to the old file and `gnus-sync-read'

2) set your gnus-sync-backend to '(lesync "url goes here") and
`gnus-sync-write'

That Should Work(tm).

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

SB> Um... could I gnus-sync-read the old Tramp+EPA file, and then
SB> gnus-sync-write the combined results into the lesync couchdb?

> 1) set your gnus-sync-backend to the old file and `gnus-sync-read'

> 2) set your gnus-sync-backend to '(lesync "url goes here") and
> `gnus-sync-write'

I didn't find `gnus-sync-write' but I found `gnus-sync-save'.  However
it didn't seem to do anything.  I even tried capturing the traffic with
wireshark, but it didn't capture anything.

Hm... maybe I need to make .newsrc.eld dirty...?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Fri, 04 Nov 2011 21:52:19 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
SB> Um... could I gnus-sync-read the old Tramp+EPA file, and then
SB> gnus-sync-write the combined results into the lesync couchdb?

>> 1) set your gnus-sync-backend to the old file and `gnus-sync-read'

>> 2) set your gnus-sync-backend to '(lesync "url goes here") and
>> `gnus-sync-write'

SB> I didn't find `gnus-sync-write' but I found `gnus-sync-save'.  However
SB> it didn't seem to do anything.  I even tried capturing the traffic with
SB> wireshark, but it didn't capture anything.

SB> Hm... maybe I need to make .newsrc.eld dirty...?

Sorry, I forgot.  To save all data unconditionally, use 
`C-u M-x gnus-sync-save RET'.  With the universal prefix it will write
everything, otherwise it filters to select only the modified groups.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Sorry, I forgot.  To save all data unconditionally, use 
> `C-u M-x gnus-sync-save RET'.  With the universal prefix it will write
> everything, otherwise it filters to select only the modified groups.

Still no luck.

Here's what I did:
 - In the *scratch* buffer do
   #+begin_src lisp
     (setq gnus-sync-backend "/ssh:sb@myhost.mydomain.com:News/gnussync")
   #+end_src
 - Start gnus with `M-x gnus RET'
 - From the *Group* buffer, do `M-x gnus-sync-read RET'
 - In the *scratch* buffer do
   #+begin_src lisp
     (setq gnus-sync-backend '("http://lesync.mydomain.com/sb"))
   #+end_src
 - From the *Group* buffer, do `C-u M-x gnus-sync-save RET'

No HTTP traffic at all, on the gnus-sync-save, according to wireshark.

Incompatible format with the "old" gnussync file...?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Sat, 05 Nov 2011 08:53:06 +0100 Steinar Bang <sb@dod.no> wrote: 

SB>      (setq gnus-sync-backend '("http://lesync.mydomain.com/sb"))

SB> No HTTP traffic at all, on the gnus-sync-save, according to wireshark.

SB> Incompatible format with the "old" gnussync file...?

No, you have the variable format wrong for LeSync:

(setq gnus-sync-backend '(lesync "http://lesync.info:5984/tzz"))

You can also `M-x customize-variable gnus-sync-backend RET'

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> No, you have the variable format wrong for LeSync:
> (setq gnus-sync-backend '(lesync "http://lesync.info:5984/tzz"))

Er, yes, sorry.  Corrected.

This gave me progress of sorts: a JSON document was POSTed, but it
failed with a "400 Bad Request", and the following message in the
minibuffer: 
 Wrong type argument: listp, "invalid UTF-8 JSON"

Googling found me these two URLs near the top:
 http://stackoverflow.com/questions/3347974/curl-giving-invalid-utf-8-json-error-from-couchdb-although-json-is-fine-any-i
 http://stackoverflow.com/questions/5342198/couchdb-view-throwing-invalid-utf-8-json-error-when-using-key-startkey

Seems to be that CouchDB wants " quotes in JSON to be escaped, ie. \".

Don't know why you're not seeing that issue?  The version of CouchDB,
perhaps? (I'm running a version prior to 1.0)

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> Don't know why you're not seeing that issue?  The version of CouchDB,
> perhaps? (I'm running a version prior to 1.0)

Or maybe you're running emacs24 with built-in http support?  And I'm
using emacs23 forking off curl in a process?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Sat, 05 Nov 2011 12:18:52 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Steinar Bang <sb@dod.no>:
>> Don't know why you're not seeing that issue?  The version of CouchDB,
>> perhaps? (I'm running a version prior to 1.0)

SB> Or maybe you're running emacs24 with built-in http support?  And I'm
SB> using emacs23 forking off curl in a process?

Yeah, that's almost definitely it.  Ugh.  The JSON data won't survive
the process call I guess.  The good news is, this is probably a bug in
Emacs 23 and we should be able to patch around it.  Can you try with
Emacs 24?

Thanks
Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Yeah, that's almost definitely it.  Ugh.  The JSON data won't survive
> the process call I guess.  The good news is, this is probably a bug in
> Emacs 23 and we should be able to patch around it.  Can you try with
> Emacs 24?

Hm... not on this little netbook (unless there's a .deb floating around
somewhere...?).  I haven't got a dev environment here (Acer Aspire One
110, 1GB RAM, 8GB SSD)).

I will see what I can find.

Re: gnus-sync.el v2

[Adam Sjøgren]

On Sat, 05 Nov 2011 18:54:17 +0100, Steinar wrote:

[Emacs 24]

> Hm... not on this little netbook (unless there's a .deb floating around
> somewhere...?).

You can find .debs for Debian (and a link to Ubuntu ones) here:

 * http://emacs.naquadah.org/


  Best regards,

    Adam

-- 
 "I always liked songs with parentheses in the title."        Adam Sjøgren
                                                         asjo@koldfront.dk

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> asjo@koldfront.dk (Adam Sjøgren):

> On Sat, 05 Nov 2011 18:54:17 +0100, Steinar wrote:
> [Emacs 24]

>> Hm... not on this little netbook (unless there's a .deb floating around
>> somewhere...?).

> You can find .debs for Debian (and a link to Ubuntu ones) here:

>  * http://emacs.naquadah.org/

Thanks!  I installed the Ubuntu 10.10 version from here
 https://launchpad.net/~cassou/+archive/emacs

I got the same issue here.  Ie. "400 Bad Request", and 
 Wrong type argument: listp, "invalid UTF-8 JSON"
in the minibuffer

`C-u M-x emacs-version' says
  GNU Emacs 24.0.91.1 (i686-pc-linux-gnu, GTK+ Version 2.22.0) of 2011-11-04 on doubah, modified by Debian

But how do I find out whether this emacs have been linked with libcurl
(I'm assuming libcurl is what's providing the HTTP support...?), or not?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Sat, 05 Nov 2011 11:45:54 +0100 Steinar Bang <sb@dod.no> wrote: 

SB> This gave me progress of sorts: a JSON document was POSTed, but it
SB> failed with a "400 Bad Request", and the following message in the
SB> minibuffer: 
SB>  Wrong type argument: listp, "invalid UTF-8 JSON"

Set `gnus-verbose' to 12 temporarily, re-run the request, and port here
or send me the *Messages* contents.  That will show the JSON document.

Also if you can send me your .newsrc.eld that would help.

I don't think it's a problem with the CouchDB version.

Thanks
Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Set `gnus-verbose' to 12 temporarily, re-run the request, and port
> here or send me the *Messages* contents.  That will show the JSON
> document.

I can do better than that.  I can send you the wireshark dump of the
HTTP session.

> Also if you can send me your .newsrc.eld that would help.

I can send you that one also on email.

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> I can do better than that.  I can send you the wireshark dump of the
> HTTP session.

Looking at that wireshark dump myself: it does look like the " quotes
are in place where they should be.

Is there a JSON syntax checker somewhere I wonder...?

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
>>>>> Steinar Bang <sb@dod.no>:

>> I can do better than that.  I can send you the wireshark dump of the
>> HTTP session.

> Looking at that wireshark dump myself: it does look like the " quotes
> are in place where they should be.

> Is there a JSON syntax checker somewhere I wonder...?

I found this one: http://jsonlint.com/

It reports: Valid JSON

Hm...

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
>>>>> Steinar Bang <sb@dod.no>:
>>>>> Steinar Bang <sb@dod.no>:

>>> I can do better than that.  I can send you the wireshark dump of the
>>> HTTP session.

>> Looking at that wireshark dump myself: it does look like the " quotes
>> are in place where they should be.

>> Is there a JSON syntax checker somewhere I wonder...?

> I found this one: http://jsonlint.com/
> It reports: Valid JSON

It could be a truncation issue.  If I look at the
/var/log/coucdb/couchdb file on the server, it contains a log line that
start with
 [Sat, 05 Nov 2011 10:23:14 GMT] [error] [<0.16092.23>] attempted upload of invalid JSON {"docs":[{"_id":...

The JSON in the log line ends abruptly.  If I dump all of the JSON in
the log line to a file, I get a file of size 154991bytes, while the
POSTed file, as extracted from the wireshark dump, is 155005 bytes.

With 155k I probably don't hit any hard limits on couchdb doc size (I
haven't checkd what they are, though).

Could it be an error in the HTTP header...?

The header says
 Content-length: 154991

I guess that explains the cutoff, but I have no idea why the count is
wrong.

Re: gnus-sync.el v2

[Adam Sjøgren]

On Sat, 05 Nov 2011 20:59:15 +0100, Steinar wrote:

> The header says
>  Content-length: 154991

> I guess that explains the cutoff, but I have no idea why the count is
> wrong.

Maybe a character-count vs. byte-count error?

Do you by any chance get something like 14 Norwegian (two-byte
utf8-)chars in the JSON-file?


  Best regards,

-- 
 "I always liked songs with parentheses in the title."        Adam Sjøgren
                                                         asjo@koldfront.dk

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> asjo@koldfront.dk (Adam Sjøgren):

> Do you by any chance get something like 14 Norwegian (two-byte
> utf8-)chars in the JSON-file?

I have no non-ascii group names, and no non-ascii server names, so I
didn't think so.

And neither the regexp, nor the find-first-non-ascii-char function in
this page, are able to find any non-ascii characters in the JSON file
extracted from the wireshark dump:
 http://www.emacswiki.org/emacs/FindingNonAsciiCharacters

I find no non-ascii characters in the .newsrc.eld either (neither by the
regexp search, nor when running the find-first-non-ascii-char function).

Re: gnus-sync.el v2

[Adam Sjøgren]

On Sat, 05 Nov 2011 23:13:55 +0100, Steinar wrote:

> I have no non-ascii group names, and no non-ascii server names, so I
> didn't think so.

Ok - it was just a guess. The lengths not matching sounds odd.


  Best regards,

    Adam

-- 
 "I always liked songs with parentheses in the title."        Adam Sjøgren
                                                         asjo@koldfront.dk

Re: gnus-sync.el v2

[Ted Zlatanov]

On Sat, 05 Nov 2011 23:13:55 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> asjo@koldfront.dk (Adam Sjøgren):
>> Do you by any chance get something like 14 Norwegian (two-byte
>> utf8-)chars in the JSON-file?

SB> I have no non-ascii group names, and no non-ascii server names, so I
SB> didn't think so.

Haha!  You do!  In the JSON:

"nnrss:NRK P1 \u00e2€“ ... n\u00c3\u00a5"

and in the newsrc you sent me: "nnrss:NRK2 \342\200\223 ..."

(I hope this is not terribly embarassing, I omitted some ASCII text from
the group names to anonymize them.)

Temporarily, you can set `gnus-sync-newsrc-groups' to just '("nntp") to
see it work.  You can add the nnrss groups harmlessly later.

I thought json.el would handle this but apparently it doesn't to
CouchDB's satisfaction; the JSON may indeed be valid.  I will try to fix
this (or someone else can do it if they feel frisky, see
`json-encode-char' and `json-read-escaped-char').

SB> I find no non-ascii characters in the .newsrc.eld either (neither by the
SB> regexp search, nor when running the find-first-non-ascii-char function).

Sorry, no idea why that didn't work.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

SB> I have no non-ascii group names, and no non-ascii server names, so I
SB> didn't think so.

> Haha!  You do!  In the JSON:

> "nnrss:NRK P1 \u00e2€“ ... n\u00c3\u00a5"

> and in the newsrc you sent me: "nnrss:NRK2 \342\200\223 ..."

Darn! :-)

> (I hope this is not terribly embarassing, I omitted some ASCII text
> from the group names to anonymize them.)

Not very embarrassing.  It was an early attempt at using nnrss to
read/listen to podcasts.

> Temporarily, you can set `gnus-sync-newsrc-groups' to just '("nntp") to
> see it work.  You can add the nnrss groups harmlessly later.

Actually I can kill those groups.  I don't use them anymore (I use my
squeezeboxen to listen to podcasts at home, and I use rhythmbox to
download and listen to them on mylaptop).

> I thought json.el would handle this but apparently it doesn't to
> CouchDB's satisfaction; the JSON may indeed be valid. 

Actually CouchDB is quite comfortable with the JSON generated, as long
as it gets all of it.  See my experiments with feeding that JSON
directly to CouchDB using curl.

Looking at the code you use to send off the http request, ie.
#+begin_src lisp
  ...
  (let ((url-request-method method)
        (url-request-extra-headers headers)
        (url-request-data (if kvdata (json-encode kvdata) nil)))
    (with-current-buffer (url-retrieve-synchronously url)
      ...
#+end_src
the setting of content-length is left to url-retrieve-synchronously, and
it is that code that fails...?

> I will try to fix this (or someone else can do it if they feel frisky,
> see `json-encode-char' and `json-read-escaped-char').

Hm... thoses are ok I think.  But it looks like Adam was right and that
url-retrieve and friends, set content-length from the number of
characters in the data they are trying to send.  And that's actually a
bug in that function, which apparently is present in emacs24 snapshots.
Content-Length should be in bytes.

I guess someone should report it...? :-)

Re: gnus-sync.el v2

[Ted Zlatanov]

On Sun, 06 Nov 2011 14:34:51 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> I thought json.el would handle this but apparently it doesn't to
>> CouchDB's satisfaction; the JSON may indeed be valid. 

SB> Actually CouchDB is quite comfortable with the JSON generated, as long
SB> as it gets all of it.  See my experiments with feeding that JSON
SB> directly to CouchDB using curl.

Oh I see.  OK.

SB> Looking at the code you use to send off the http request, ie.
SB> #+begin_src lisp
SB>   ...
SB>   (let ((url-request-method method)
SB>         (url-request-extra-headers headers)
SB>         (url-request-data (if kvdata (json-encode kvdata) nil)))
SB>     (with-current-buffer (url-retrieve-synchronously url)
SB>       ...
SB> #+end_src
SB> the setting of content-length is left to url-retrieve-synchronously, and
SB> it is that code that fails...?

>> I will try to fix this (or someone else can do it if they feel frisky,
>> see `json-encode-char' and `json-read-escaped-char').

SB> Hm... thoses are ok I think.  But it looks like Adam was right and that
SB> url-retrieve and friends, set content-length from the number of
SB> characters in the data they are trying to send.  And that's actually a
SB> bug in that function, which apparently is present in emacs24 snapshots.
SB> Content-Length should be in bytes.

SB> I guess someone should report it...? :-)

That's work!  No way!

Oh, OK.  `url-http-create-request' says:

             (if url-http-data
                 (concat
                  "Content-length: " (number-to-string
                                      (length url-http-data))
                  "\r\n"))

which seems reasonable to me (the JSON data at that point contains only
the escaped characters).  Can you log `url-http-data' at that point for
your case?  Or maybe make a test case that fails?  I won't be able to
look at this until later, and a reproducible test case is the nicest way
to submit a bug report.

Thanks
Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:
> On Sun, 06 Nov 2011 14:34:51 +0100 Steinar Bang <sb@dod.no> wrote: 

> Oh, OK.  `url-http-create-request' says:

>              (if url-http-data
>                  (concat
>                   "Content-length: " (number-to-string
>                                       (length url-http-data))
>                   "\r\n"))

> which seems reasonable to me (the JSON data at that point contains only
> the escaped characters).

`C-h f length RET' says:

 length is a built-in function in `C source code'.

 (length sequence)

 Return the length of vector, list or string sequence.
 A byte-code function object is also allowed.
 If the string contains multibyte characters, this is not necessarily
 the number of bytes in the string; it is the number of characters.
 To get the number of bytes, use `string-bytes'.

so, it probably should have been 

 (if url-http-data
     (concat
      "Content-length: " (number-to-string
                          (number-to-string url-http-data))
      "\r\n"))

if url-http-data always is a string.

If it is any kind of sequence, I guess it would have to be something
like:

 (if url-http-data
     (let (content-length (number-to-string
                           (if (stringp url-http-data)
                               (string-bytes url-http-data)
                             (length url-http-data))))
     (concat "Content-length: " content-length "\r\n")))

(my only lisp experience comes from my .emacs and .gnus.el files... but
something like this, I guess...?)

> Can you log `url-http-data' at that point for your case?

Not easily, I think... that would require building emacs from source,
and modifying the lisp source...?

> Or maybe make a test case that fails? 

Even less easily.  I have no idea where the lisp unit tests are, and
what they can do

> I won't be able to look at this until later, and a reproducible test
> case is the nicest way to submit a bug report.

Yeah, I know.  We'll see.

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> so, it probably should have been 

>  (if url-http-data
>      (concat
>       "Content-length: " (number-to-string
>                           (number-to-string url-http-data))
>       "\r\n"))

> if url-http-data always is a string.

No, it shouldn't.  It should have been:

 (if url-http-data
     (concat
      "Content-length: " (number-to-string
                          (string-bytes url-http-data))
      "\r\n"))

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> Actually I can kill those groups.  I don't use them anymore (I use my
> squeezeboxen to listen to podcasts at home, and I use rhythmbox to
> download and listen to them on mylaptop).

FWIW, with the troublesome nnrss groups gone from my .newsrc.eld, I
could load the old gnussync file, and save it to lesync database.

So what's left now is to start using it.

Next step in the config, is:
 ;; - use gnus-after-set-mark-hook and gnus-before-update-mark-hook to
 ;;   catch the mark updates

I guess there are functions that should be added to these hooks, but which
ones?  Or am I supposed to write my own functions to add to the hooks?

Re: gnus-sync.el v2

[Steinar Bang]

Moving on, I have started to use the new gnus-sync v2 using lesync.

I am manually using `M-x gnus-sync-read RET' to get updates and manually
using `M-x gnus-sync-save RET' to save my marks.

A little bit of strangeness so far: it asked me if I wanted to create a
group with a GUID-ish name, and it tried to create a new topic and
failed with some error message (that didn't appear in the *Messages*
buffer, it looks like).  Here's the end of the *Messages* buffer from
the gnus-sync-read (I'm not showing all of the groups, there seemed to
be a message like that for all groups).

...
gnus-sync-lesync-post-save-group-entry: use `gnus-sync-read' to resolve the conflict synchronizing nntp+news.gmane.org:gmane.comp.gnu.tuxpaint.devel.rubberstamps to http://lesync.mydomain.com/sb: Document update conflict.
Contacting host: lesync.mydomain.com:80
Subscribe to group 8363dd377dc7d1acc19dca587800023b?(y or n) 
Delete group 8363dd377dc7d1acc19dca587800023b from server?(y or n) 
Contacting host: lesync.mydomain.com:80
Reading [text/plain;charset=utf-8]... 96 bytes of 95 bytes (101%)
gnus-sync-lesync-read-group-entry: removed group 8363dd377dc7d1acc19dca587800023b from server http://lesync.mydomain.com/sb
gnus-sync-lesync-read-group-entry: ignoring entry 8363dd377dc7d1acc19dca587800023b, it wasn't subscribed.  Call `gnus-sync-read' with C-u to force it.
gnus-sync-lesync-read-group-entry: ignoring entry 8363dd377dc7d1acc19dca587800023b, aa1steinar not in sources ""
gnus-sync-save: nothing to save to the LeSync backend

Re: gnus-sync.el v2

[Steinar Bang]

It also seems like my database was created without a password.

Where does gnus-sync get the password from?  Does it prompt the user?
Does it save it in the auth-thingy?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Mon, 07 Nov 2011 21:08:08 +0100 Steinar Bang <sb@dod.no> wrote: 

SB> It also seems like my database was created without a password.
SB> Where does gnus-sync get the password from?  Does it prompt the user?
SB> Does it save it in the auth-thingy?

From the inline docs:

You need to set up a server admin:

;; 1. install CouchDB, set up a real server admin user, and create a
;; database, e.g. "tzz" and save the URL,
;; e.g. http://lesync.info:5984/tzz

and then

;; 2. run `M-: (gnus-sync-lesync-setup "http://lesync.info:5984/tzz" "tzzadmin" "mypassword" "mysalt" t t)'

;;    (If you run it more than once, you have to remove the entry from
;;    _users yourself.  This is intentional.  This sets up a database
;;    admin for the "tzz" database, distinct from the server admin
;;    user in (1) above.)

This will create a user "tzzadmin" that is an admin for the "tzz"
database.

Look in http://lesync.info:5984/_utils/database.html?_users to look at
the users in Futon.

Again in Futon, look at a database and click on the "security" button on
top to adjust the readers/admins list.  `gnus-sync-lesync-setup' will
have already modified it for you.

I didn't put this in the gnus-sync.el docs because it's really ancillary
but I guess it will trip people up over time.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:


> On Mon, 07 Nov 2011 21:08:08 +0100 Steinar Bang <sb@dod.no> wrote: 
[snip!]
SB> Where does gnus-sync get the password from?  Does it prompt the user?
SB> Does it save it in the auth-thingy?

Note: this was more about what gnus-sync does, when confronted with a
401 Authorization Required

Ie. does it request a username/password from the user? And does it
persist the username/password combination?

> From the inline docs:
> You need to set up a server admin:

> ;; 1. install CouchDB, set up a real server admin user, and create a
> ;; database, e.g. "tzz" and save the URL,
> ;; e.g. http://lesync.info:5984/tzz

I know.

I tried to do that, based on the hints in the debian couchdb config
files: in the /etc/couchdb/local.ini file uncommented the [admins]
section and added a username = password line, and then restarted
couchdb.

> and then

> ;; 2. run `M-: (gnus-sync-lesync-setup "http://lesync.info:5984/tzz" "tzzadmin" "mypassword" "mysalt" t t)'

Did that one as well, with the URL, tzzadmin, mypassword, and mysalt
appropriately modified.

> ;;    (If you run it more than once, you have to remove the entry from
> ;;    _users yourself.

That's why I didn't do it more than once.

>  This is intentional.  This sets up a database
> ;;    admin for the "tzz" database, distinct from the server admin
> ;;    user in (1) above.)

> This will create a user "tzzadmin" that is an admin for the "tzz"
> database.

> Look in http://lesync.info:5984/_utils/database.html?_users to look at
> the users in Futon.

Got two entries here.  One is the one created by the
gnus-sync-lesync-setup function, and the other one is called
_design/auth.

> Again in Futon, look at a database and click on the "security" button on
> top to adjust the readers/admins list.  `gnus-sync-lesync-setup' will
> have already modified it for you.

I have:
 Admins
  Names: ["myadmin"]
  Roles: []

 Readers
  Names: ["myadmin"]
  Roles: []

> I didn't put this in the gnus-sync.el docs because it's really ancillary
> but I guess it will trip people up over time.

It could be that my real admin created in the /etc/couchdb/local.ini
file hasn't been picked up by couchdb...?

Re: gnus-sync.el v2

[Ted Zlatanov]

On Wed, 09 Nov 2011 10:27:20 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> On Mon, 07 Nov 2011 21:08:08 +0100 Steinar Bang <sb@dod.no> wrote: 
SB> [snip!]
SB> Where does gnus-sync get the password from?  Does it prompt the user?
SB> Does it save it in the auth-thingy?

SB> Note: this was more about what gnus-sync does, when confronted with a
SB> 401 Authorization Required

SB> Ie. does it request a username/password from the user? And does it
SB> persist the username/password combination?

That's handled by the URL package.  I believe you just add an entry to
.authinfo:

machine lesync.info:5984 port http login mylogin password mypassword

At least for me this works, I don't have to enter a password.  But you
can also enter the password manually once per Emacs session.

>> From the inline docs:
>> You need to set up a server admin:

>> ;; 1. install CouchDB, set up a real server admin user, and create a
>> ;; database, e.g. "tzz" and save the URL,
>> ;; e.g. http://lesync.info:5984/tzz

SB> I know.

SB> I tried to do that, based on the hints in the debian couchdb config
SB> files: in the /etc/couchdb/local.ini file uncommented the [admins]
SB> section and added a username = password line, and then restarted
SB> couchdb.

I did it from Futon.  Dunno if local.ini works the same way.

>> Look in http://lesync.info:5984/_utils/database.html?_users to look at
>> the users in Futon.

SB> Got two entries here.  One is the one created by the
SB> gnus-sync-lesync-setup function, and the other one is called
SB> _design/auth.

Yup.

>> Again in Futon, look at a database and click on the "security" button on
>> top to adjust the readers/admins list.  `gnus-sync-lesync-setup' will
>> have already modified it for you.

SB> I have:
SB>  Admins
SB>   Names: ["myadmin"]
SB>   Roles: []

SB>  Readers
SB>   Names: ["myadmin"]
SB>   Roles: []

>> I didn't put this in the gnus-sync.el docs because it's really ancillary
>> but I guess it will trip people up over time.

SB> It could be that my real admin created in the /etc/couchdb/local.ini
SB> file hasn't been picked up by couchdb...?

I think the DB admin is separate from the table admin list.  But I
honestly don't know for sure.

Ted

Re: gnus-sync.el v2

[Ted Zlatanov]

On Mon, 07 Nov 2011 20:55:31 +0100 Steinar Bang <sb@dod.no> wrote: 

SB> Moving on, I have started to use the new gnus-sync v2 using lesync.
SB> I am manually using `M-x gnus-sync-read RET' to get updates and manually
SB> using `M-x gnus-sync-save RET' to save my marks.

SB> A little bit of strangeness so far: it asked me if I wanted to create a
SB> group with a GUID-ish name, and it tried to create a new topic and
SB> failed with some error message (that didn't appear in the *Messages*
SB> buffer, it looks like).  Here's the end of the *Messages* buffer from
SB> the gnus-sync-read (I'm not showing all of the groups, there seemed to
SB> be a message like that for all groups).

SB> ...
SB> gnus-sync-lesync-post-save-group-entry: use `gnus-sync-read' to
SB> resolve the conflict synchronizing
SB> nntp+news.gmane.org:gmane.comp.gnu.tuxpaint.devel.rubberstamps to
SB> http://lesync.mydomain.com/sb: Document update conflict.
SB> Contacting host: lesync.mydomain.com:80
SB> Subscribe to group 8363dd377dc7d1acc19dca587800023b?(y or n) 
SB> Delete group 8363dd377dc7d1acc19dca587800023b from server?(y or n) 
SB> Contacting host: lesync.mydomain.com:80
SB> Reading [text/plain;charset=utf-8]... 96 bytes of 95 bytes (101%)
SB> gnus-sync-lesync-read-group-entry: removed group 8363dd377dc7d1acc19dca587800023b from server http://lesync.mydomain.com/sb
SB> gnus-sync-lesync-read-group-entry: ignoring entry 8363dd377dc7d1acc19dca587800023b, it wasn't subscribed.  Call `gnus-sync-read' with C-u to force it.
SB> gnus-sync-lesync-read-group-entry: ignoring entry 8363dd377dc7d1acc19dca587800023b, aa1steinar not in sources ""
SB> gnus-sync-save: nothing to save to the LeSync backend

That's weird.  Use Futon (http://lesync.mydomain.com/_utils) to see and
possibly remove that entry.  I don't know how it got created.  Perhaps
it was a consequence of the broken JSON post.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> That's weird.  Use Futon (http://lesync.mydomain.com/_utils) to see
> and possibly remove that entry.  I don't know how it got created.
> Perhaps it was a consequence of the broken JSON post.

Maybe the JSON has autocorrected.  I haven't seen that one for a while.

My main problem now is that gnus' subscription to a group isn't
persisted.  I get these questions every time I start up a new emacs
(note: not a new gnus, but a new emacs process).

Messages like these:
 Read group nntp+news.opera.com:opera.beta even though somehost.myjob.com is not in sources ("aa1steinar")?(y or n) 

Re: gnus-sync.el v2

[Ted Zlatanov]

On Wed, 09 Nov 2011 10:31:05 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> That's weird.  Use Futon (http://lesync.mydomain.com/_utils) to see
>> and possibly remove that entry.  I don't know how it got created.
>> Perhaps it was a consequence of the broken JSON post.

SB> Maybe the JSON has autocorrected.  I haven't seen that one for a while.

SB> My main problem now is that gnus' subscription to a group isn't
SB> persisted.  I get these questions every time I start up a new emacs
SB> (note: not a new gnus, but a new emacs process).

SB> Messages like these:
SB>  Read group nntp+news.opera.com:opera.beta even though somehost.myjob.com is not in sources ("aa1steinar")?(y or n) 

`gnus-sync-lesync-name' needs to be in that list (it defaults to
(system-name)).  Do `C-u M-x gnus-sync-save' and it will add the current
host name to the list of sources.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> `gnus-sync-lesync-name' needs to be in that list (it defaults to
> (system-name)).  Do `C-u M-x gnus-sync-save' and it will add the
> current host name to the list of sources.

Ok, that helped for those groups I do want synced.  But I am prompted
on every start, about the groups I don't want synced.

What to do,...?  I guess I could try deleting those groups on the
server, using futon...?  I will try that.

Re: gnus-sync.el v2

[Ted Zlatanov]

On Sun, 06 Nov 2011 22:29:22 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Steinar Bang <sb@dod.no>:
>> Actually I can kill those groups.  I don't use them anymore (I use my
>> squeezeboxen to listen to podcasts at home, and I use rhythmbox to
>> download and listen to them on mylaptop).

SB> FWIW, with the troublesome nnrss groups gone from my .newsrc.eld, I
SB> could load the old gnussync file, and save it to lesync database.

SB> So what's left now is to start using it.

SB> Next step in the config, is:
SB>  ;; - use gnus-after-set-mark-hook and gnus-before-update-mark-hook to
SB>  ;;   catch the mark updates

SB> I guess there are functions that should be added to these hooks, but which
SB> ones?  Or am I supposed to write my own functions to add to the hooks?

No, just read and save, you're done :)

What you show is the TODO list.

Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> No, just read and save, you're done :)

Yes, I figured it out eventually.

Frequent manual `M-x gnus-sync-save' is quite doable, since it's so
fast.

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

[snip!]
> The JSON in the log line ends abruptly.  If I dump all of the JSON in
> the log line to a file, I get a file of size 154991bytes, while the
> POSTed file, as extracted from the wireshark dump, is 155005 bytes.

> With 155k I probably don't hit any hard limits on couchdb doc size (I
> haven't checkd what they are, though).

> Could it be an error in the HTTP header...?

> The header says
>  Content-length: 154991

> I guess that explains the cutoff,

Here's also a difference from Teds setup: I'm running coucdb behind a
reverse proxy apache.  It could be that apache is stricter wrt
content-length than couchdb...?  (if so, that's a good thing when
talking to the internet... expect that a client lying to you, is up to
nothing good...).

I tried one experiment, which was to copy the JSON file I extracted from
the wireshark dump over to the server, and send it to couchdb, with the
command: 
 curl --data-binary @/tmp/lesync_sync.json --header "Content-Type: application/json" http://localhost:5984/sb

This was successful.  But a wireshark dump of the HTTP traffic on port
5984, showed that the content-length of the request was set to the
correct 155005 bytes value.

So this didn't prove anything other than that the lesync post from gnus
have a wrong content-length...

(Also, the wireshark dump showed that curl's POST request had an
 Expect: 100-Continue
header, and that the client politely waited for a "100 Continue"
response before sending the JSON in the request body to the server.)

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Then set it up, gnus-sync.el has a function for that too.  I gave
> examples in the commentary.  Read it carefully.  It should be really
> easy.

I'm a little confused anyway...:-)

I the "admin user" in 1. and 2. the same?  Or does "real admin user"
mean the user couchdb is running as (on debian by default the user
named... "couchdb")?

 ;; 1. install CouchDB, set up a real admin user, and create a
 ;; database, e.g. "tzz" and save the URL,
 ;; e.g. http://lesync.info:5984/tzz

 ;; 2. run `M-: (gnus-sync-lesync-setup "http://lesync.info:5984/tzz" "tzzadmin" "mypassword" "mysalt" t t)'
 ;;    (If you run it more than once, you have to remove the entry from
 ;;    _users yourself.  This is intentional.)

Re: gnus-sync.el v2

[Ted Zlatanov]

On Sat, 29 Oct 2011 21:02:37 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> Then set it up, gnus-sync.el has a function for that too.  I gave
>> examples in the commentary.  Read it carefully.  It should be really
>> easy.

SB> I'm a little confused anyway...:-)

SB> I the "admin user" in 1. and 2. the same?  Or does "real admin user"
SB> mean the user couchdb is running as (on debian by default the user
SB> named... "couchdb")?

SB>  ;; 1. install CouchDB, set up a real admin user, and create a
SB>  ;; database, e.g. "tzz" and save the URL,
SB>  ;; e.g. http://lesync.info:5984/tzz

SB>  ;; 2. run `M-: (gnus-sync-lesync-setup "http://lesync.info:5984/tzz" "tzzadmin" "mypassword" "mysalt" t t)'
SB>  ;;    (If you run it more than once, you have to remove the entry from
SB>  ;;    _users yourself.  This is intentional.)

There are two kinds of admins, server admins and DB admins (the
distinction is based on the _users definition of the user).  Server
admins can create databases.

I believe `gnus-sync-lesync-setup' creates a DB admin for the "tzz"
database, not a server admin.

None of these admin types are connected to the Unix user name, all the
CouchDB users are entirely virtual and defined in the _users database.

When you install CouchDB, anyone is a server admin until you define at
least one valid one.  So you should do that.  That's what (1) means.
I'll correct the docs when I get a chance.

Thanks
Ted

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>> Whether that's OK for everyone I can't say; they can always run
>> things through a proxy or VPN or whatever makes sense.  For me it's
>> good enough.

> I'll do a little bit of research on what's possible.

Found this: How to put CouchDB behind proxy (for access control and/or high availability)
 http://wiki.apache.org/couchdb/Apache_As_a_Reverse_Proxy

(If we're using port 80 or port 443 I'll have to do something like this
anyway, because apache owns those ports)

Re: gnus-sync.el v2

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I'm not sure if that applies to Gnus; I think if Yamaoka-san does not
> synchronize gnus-sync.el with the Emacs trunk, that's sufficient to make
> us compliant with the Emacs feature freeze.

I think No Gnus should remain in a feature freeze for the time being.
There's still Emacs 24 issues to be shaken out...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I'm not sure if that applies to Gnus; I think if Yamaoka-san does not
>> synchronize gnus-sync.el with the Emacs trunk, that's sufficient to make
>> us compliant with the Emacs feature freeze.

> I think No Gnus should remain in a feature freeze for the time being.
> There's still Emacs 24 issues to be shaken out...

Yes, but the file that Ted updated isn't synced to emacs24 by
Yamoka-san.

Re: gnus-sync.el v2

[Ted Zlatanov]

On Thu, 03 Nov 2011 23:43:33 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I'm not sure if that applies to Gnus; I think if Yamaoka-san does not
>> synchronize gnus-sync.el with the Emacs trunk, that's sufficient to make
>> us compliant with the Emacs feature freeze.

LMI> I think No Gnus should remain in a feature freeze for the time being.
LMI> There's still Emacs 24 issues to be shaken out...

<larsi>Haha!  Too late!  I've done it!</larsi>

Ted

Re: gnus-sync.el v2

[Andreas Schwab]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Regarding gnus-sync.el, the good news is that it works great,

It cannot even read back what it has saved.

Wrong type argument: listp, 6

Also, it no longer initializes when loaded.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: gnus-sync.el v2

[Andreas Schwab]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> Regarding gnus-sync.el, the good news is that it works great,
>
> It cannot even read back what it has saved.
>
> Wrong type argument: listp, 6

Don't you think it is a stupid idea to change the format of the stored
data without making it backward and forward compatible?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: gnus-sync.el v2

[Ted Zlatanov]

On Sun, 02 Dec 2012 11:19:18 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Andreas Schwab <schwab@linux-m68k.org> writes:
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>> 
>>> Regarding gnus-sync.el, the good news is that it works great,
>> 
>> It cannot even read back what it has saved.
>> 
>> Wrong type argument: listp, 6

AS> Don't you think it is a stupid idea to change the format of the stored
AS> data without making it backward and forward compatible?

Absolutely, and if that's happened, I apologize.  But I don't recall
doing so.  I use gnus-sync daily and was not aware of this bug or design
flaw.  Can you post the backtrace and possibly the data that triggered it?

I do (really really) want to rewrite gnus-sync.el to use native IMAP
commands instead of an external CouchDB installation, and at that time
will settle on a new, more robust format for flag and general data
storage.  I was supposed to get to that last month but my work
commitment was extended by a few weeks; the rewrite is at the top of my
TODO list.

I also asked Lars for help with the IMAP side but he hasn't been
available.  If you or anyone else wants to help, I need generic
functions that, given an IMAP mailbox, a string key, and an alist, can
store the alist under the key or delete the key (so basically I want a
hashtable implemented in an IMAP mailbox).  Writing those functions will
probably be 80-90% of the time I spend on the rewrite, so any help is
appreciated.  I used to have some understanding of imap.el, but the new
nnimap.el is a super wonderful mystery to me.

Ted

Re: gnus-sync.el v2

[Ted Zlatanov]

On Sun, 02 Dec 2012 11:17:10 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I do (really really) want to rewrite gnus-sync.el to use native IMAP
TZ> commands instead of an external CouchDB installation, and at that time
TZ> will settle on a new, more robust format for flag and general data
TZ> storage.  I was supposed to get to that last month but my work
TZ> commitment was extended by a few weeks; the rewrite is at the top of my
TZ> TODO list.

TZ> I also asked Lars for help with the IMAP side but he hasn't been
TZ> available.  If you or anyone else wants to help, I need generic
TZ> functions that, given an IMAP mailbox, a string key, and an alist, can
TZ> store the alist under the key or delete the key (so basically I want a
TZ> hashtable implemented in an IMAP mailbox).  Writing those functions will
TZ> probably be 80-90% of the time I spend on the rewrite, so any help is
TZ> appreciated.  I used to have some understanding of imap.el, but the new
TZ> nnimap.el is a super wonderful mystery to me.

I spent time prototyping and thinking about this.

I still think the right approach is to use IMAP and nnimap.el for
recording the data, as Lars suggested originally.

I have realized that the whole gnus-sync.el structure is unnecessary.
Gnus already has synchronization built-in.  The dribble file is perfect
for synchronizing the data, and all the Gnus internals already write
group info and topics to the dribble file.

To implement a new IMAP-synchronized mode, we need to make every Gnus
instance behave like a slave Gnus, always consulting the dribble IMAP
group and saving back to it.

The configuration will be to simply set the dribble filename to an IMAP
folder and to set some gnus-sync variable to indicate perpetual slavery.

This has to be done in profiles, so you can have multiple connectivity
profiles but still share the marks and other information.

The format for the IMAP dribble group could be a series of messages in
the following formats:

1. set group info
subject = "gnus-group-set-info GROUPNAME"
from = PROFILE
body = (gnus-group-set-info ...)

2. set topology
subject = "setq gnus-topic-topology"
from = PROFILE
body = (setq gnus-topic-topology ...)

3. set group/topic parameters
subject = "gnus-topic-set-parameters GROUP-OR-TOPIC"
from = PROFILE
body = (gnus-topic-set-parameters ...)

4. set server parameters
subject = "PROFILE gnus-server-set-info SERVER"
from = PROFILE
body = (gnus-server-set-info ...)

All of the above are already entered into the dribble file at the right
time, with the right data, except for the profiles.

The sequence for saving each of the types above, given SUBJECT and BODY
and synchronization folder SYNC-FOLDER, and local connectivity profile
PROFILE:

1. create a new message with SUBJECT and BODY in SYNC-FOLDER

2. mark as 'read' all older messages with the same subject in FOLDER

Note that (2) must wait for (1) to succeed, but operates on all the
messages with SUBJECT except the newest one, so it's idempotent and will
not break if multiple clients do it at the same time.  Also, (2) can
fail and should be sent without waiting for a confirmation back.  It's
intended to speed up the read operations, but doesn't matter for the
save operations.

The sequence to synchronize:

1. find all the unread messages in SYNC-FOLDER.  Remember them in an
alist with cell format (SUBJECT . MESSAGE-ID).  This will be a small
alist.  For each distinct SUBJECT, only remember and load the latest
message.

2. ask the IMAP server to notify us when SYNC-FOLDER's UID changes and
do the usual dribble loading of new messages, skipping any message IDs
we have already remembered.  Are these notifications possible?

We should NOT synchronize the currently entered group; I think to avoid
user confusion we should simply ignore changes to it with a warning, and
save it on exit in the usual way.

Note that all the operations above for reading and saving are fast on
IMAP and are idempotent.

In synchronized mode, the newsrc file can still be saved and read, but
it will be overridden by whatever is in the SYNC-FOLDER.  So it's more
of a backup.

Let me know what you think about this proposal.  I think it's the right
solution, technically and practically.  Removing code (in this case
gnus-sync.el) in favor of the existing facilities is IMHO a good way to
simplify the user experience, reduce the maintenance burden, and improve
Gnus as a whole.

Ted

Re: gnus-sync.el v2

[Eric S Fraga]

Ted Zlatanov <tzz@lifelogs.com> writes:

[...]

> Note that all the operations above for reading and saving are fast on
> IMAP and are idempotent.

Ummm see below re: "fast"

> Let me know what you think about this proposal.  I think it's the right
> solution, technically and practically.  Removing code (in this case
> gnus-sync.el) in favor of the existing facilities is IMHO a good way to
> simplify the user experience, reduce the maintenance burden, and improve
> Gnus as a whole.

Ted,

the proposal sounds fine in principle.  I only have one worry and that
worry comes from my having to use Microsoft's outlook servers for all
work emails.  My IMAP mail server sits on xxxx.outlook.com and this
server is not "fast", gnus times out on many operations with that
server, and gnus seems to hang on message retrieval more often than
not.  For instance, I posted to this group a week or two ago about
problems expiring emails from a large group: gnus gives up for some
reason after a few minutes.

Now, it could be that the solution would be to address how gnus
interacts with Microsoft's own (broken?) systems on outlook.com but I
have no idea what is needed or even if any improvements are possible.

Adding another not very robust IMAP access to the system could drive me
bonkers.  But I would say that I would hate to see my problems with MS
and outlook.com stop what appears to be a very reasonable solution being
implemented so feel free to ignore my diatribe above (a diatribe
directed really at Microsoft... gnus itself is brilliant!).

thanks,
eric

PS - any hints on how I can tweak gnus to work better with outlook would
be welcome.
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_7.9.3d-874-gf11acc

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Let me know what you think about this proposal.

Looks good to me!  I will try it out, once you have something in place. :-)

> I think it's the right solution, technically and practically.
> Removing code (in this case gnus-sync.el) in favor of the existing
> facilities is IMHO a good way to simplify the user experience, reduce
> the maintenance burden, and improve Gnus as a whole.

I agree.

Re: gnus-sync.el v2

[Steinar Bang]

>>>>> Andreas Schwab <schwab@linux-m68k.org>:
> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Regarding gnus-sync.el, the good news is that it works great,

> It cannot even read back what it has saved.

> Wrong type argument: listp, 6

Hm... do you have a default news server?  Ie. set with
gnus-select-method?

I get this error from time to time, and what I do then, is just to
delete the single group I still have on that server.

Re: gnus-sync.el v2

[Matt Ford]

Hi,

The lisp stuff I fixed for file based syncing.  See:

http://permalink.gmane.org/gmane.emacs.gnus.general/82033

Not sure what was done with it.

Cheers,
-- 
Matt

how do I subscribe to a group as a string from code?

[Ted Zlatanov]

(resending, original had "subscribe" at the beginning of a line and got
stalled, so ding-owner@lists.math.uh.edu can disregard it--thanks!)

I need this for my gnus-sync.el work.  I have save and read working, now
I want to be able to sync subscriptions.  As soon as that's working I'll
put it out.

Given a string: "nntp+news.gwene.org:gwene.org.slashdot" I want to make
a subscription to that group.  The server is not necessarily defined,
but it's OK to fail in that case.

I looked at `gnus-browse-unsubscribe-group'.  It's very tightly tied to
the server browsing buffer.  I can dig into the backend functionality
more, but was hoping someone could save me a few hours of poking around.

Thanks
Ted