'g' : get-new-news and fetch articles with agent and other agent problems

'g' : get-new-news and fetch articles with agent and other agent problems

[Xavier Maillard]

Hi,

Is there an easy way to tell gnus and agent to fetch all articles (respecting
the predicates) when getting new news ?

My setup is composed of IMAp and NNTP groups only. So I want to have all my data
locally when pressing 'g'.

Any solution ?

Another bad thing occured too. I have set 'nnmail-use-long-names' to t and since
then, simply entering take a lonnnggg time (ie, it reloads the whole active file
or whatever from the server). I have moved all my old directory to the new one
replacing all '/' by '.' and Gnus seems to be blind  behind this change even
after an gnus-agent-regenerate-groups.

Regards,

zeDek

Re: 'g' : get-new-news and fetch articles with agent and other

[Kevin Greiner]

Xavier Maillard <zedek@gnu-rox.org> writes:

> Hi,
>
> Is there an easy way to tell gnus and agent to fetch all articles (respecting
> the predicates) when getting new news ?
>
> My setup is composed of IMAp and NNTP groups only. So I want to have all my data
> locally when pressing 'g'.
>
> Any solution ?

gnus-agent-fetch-session ('J s' in the group buffer) and
gnus-agent-fetch-group ('J S' in the group buffer) offer the
functionality that you are requesting.  You should be able to create a
hook function that, when added to gnus-after-getting-new-news-hook,
will invoke the appropriate fetch function.

> Another bad thing occured too. I have set 'nnmail-use-long-names' to t and since
> then, simply entering take a lonnnggg time (ie, it reloads the whole active file
> or whatever from the server). I have moved all my old directory to the new one
> replacing all '/' by '.' and Gnus seems to be blind  behind this change even
> after an gnus-agent-regenerate-groups.

nnagent uses nnmail to read offline articles hence the dependency on
nnmail-use-long-file-names.  Changing the value of
nnmail-use-long-file-names is certainly a bad thing to do in that
gnus-agent will not be able to find anything that it previously
fetched.  However, it should work if you either deleted the news/agent
hierarchy or, as you indicated, manually renamed the paths.  Have you
tried something like:

(let ((nnmail-use-long-file-names nil))
(gnus-agent-group-pathname "nntp+news.gmane.org:gmane.emacs.gnus.general"))

and

(let ((nnmail-use-long-file-names t))
(gnus-agent-group-pathname "nntp+news.gmane.org:gmane.emacs.gnus.general"))

to compare the impact of nnmail-use-long-file-names on the agent
directory structure?

By the way, is the performance impact on imap groups, nntp groups, or
both?

Kevin

Re: 'g' : get-new-news and fetch articles with agent and other

[Xavier Maillard]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

>> Any solution ?
>
> gnus-agent-fetch-session ('J s' in the group buffer) and
> gnus-agent-fetch-group ('J S' in the group buffer) offer the
> functionality that you are requesting.  You should be able to create a
> hook function that, when added to gnus-after-getting-new-news-hook,
> will invoke the appropriate fetch function.

Yup done. Quite easy ;)

>> Another bad thing occured too. I have set 'nnmail-use-long-names' to t and since
>> then, simply entering take a lonnnggg time (ie, it reloads the whole active file
>> or whatever from the server). I have moved all my old directory to the new one
>> replacing all '/' by '.' and Gnus seems to be blind  behind this change even
>> after an gnus-agent-regenerate-groups.
>
> nnagent uses nnmail to read offline articles hence the dependency on
> nnmail-use-long-file-names.  Changing the value of

Yup saw it reading the code.

> nnmail-use-long-file-names is certainly a bad thing to do in that
> gnus-agent will not be able to find anything that it previously
> fetched.  However, it should work if you either deleted the news/agent
> hierarchy or, as you indicated, manually renamed the paths.  Have you

What do you mean by that ? 

> By the way, is the performance impact on imap groups, nntp groups, or
> both?

In fact, performance are getting worst. Entering a NNTP group is so
long I could take a cup a coffee each time I enter one. Dunno though if
it is related to my changes :) What I think is that Gnus is totally
lost whith these changes. Even with regeneration of agent groups
doesn't help. I have articles fetched that are displaid as not and vice
versa. Quite weird.

Not that performance only aplly for nntp groups since I recently
switched to nnimap.

I have other problems with IMAP though.

I told gnus to split my mail from 'INBOX'. I have a default rule (sort
of catch-all group) named INBOX.mail in which I should see all mails
that didn't match to previous rules. My problem is that the 'INBOX'
group contain plenty of articles in it and numerous of them are only
contained in this group. I thought the INBOX group would be empty. Am I
wrong with it ?

zeDek
-- 
 In Gruuik we trust

Re: 'g' : get-new-news and fetch articles with agent and other

[Kevin Greiner]

Xavier Maillard <zedek@gnu-rox.org> writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>>> Any solution ?
>>
>> gnus-agent-fetch-session ('J s' in the group buffer) and
>> gnus-agent-fetch-group ('J S' in the group buffer) offer the
>> functionality that you are requesting.  You should be able to create a
>> hook function that, when added to gnus-after-getting-new-news-hook,
>> will invoke the appropriate fetch function.
>
> Yup done. Quite easy ;)
>
>>> Another bad thing occured too. I have set 'nnmail-use-long-names' to t and since
>>> then, simply entering take a lonnnggg time (ie, it reloads the whole active file
>>> or whatever from the server). I have moved all my old directory to the new one
>>> replacing all '/' by '.' and Gnus seems to be blind  behind this change even
>>> after an gnus-agent-regenerate-groups.
>>
>> nnagent uses nnmail to read offline articles hence the dependency on
>> nnmail-use-long-file-names.  Changing the value of
>
> Yup saw it reading the code.
>
>> nnmail-use-long-file-names is certainly a bad thing to do in that
>> gnus-agent will not be able to find anything that it previously
>> fetched.  However, it should work if you either deleted the news/agent
>> hierarchy or, as you indicated, manually renamed the paths.  Have you
>
> What do you mean by that ? 

The agent has no way to know that nnmail-use-long-file-names changed
value.  Therefore, if you use the agent with
nnmail-use-long-file-names set to nil then change it to non-nil, the
agent will not be able to find any of the content fetched before
changing nnmail-use-long-file-names.

>> By the way, is the performance impact on imap groups, nntp groups, or
>> both?
>
> In fact, performance are getting worst. Entering a NNTP group is so
> long I could take a cup a coffee each time I enter one. Dunno though if
> it is related to my changes :) What I think is that Gnus is totally
> lost whith these changes. Even with regeneration of agent groups
> doesn't help. I have articles fetched that are displaid as not and vice
> versa. Quite weird.
>
> Not that performance only aplly for nntp groups since I recently
> switched to nnimap.

I'll see if I can reproduce the problem here.

> I have other problems with IMAP though.
>
> I told gnus to split my mail from 'INBOX'. I have a default rule (sort
> of catch-all group) named INBOX.mail in which I should see all mails
> that didn't match to previous rules. My problem is that the 'INBOX'
> group contain plenty of articles in it and numerous of them are only
> contained in this group. I thought the INBOX group would be empty. Am I
> wrong with it ?

Sorry, you've already exceeded my knowledge of nnimap.

Kevin

Re: 'g' : get-new-news and fetch articles with agent and other

[Xavier Maillard]

Kevin Greiner <kgreiner <at> xpediantsolutions.com> writes:

> > What do you mean by that ? 
> 
> The agent has no way to know that nnmail-use-long-file-names changed
> value.  Therefore, if you use the agent with
> nnmail-use-long-file-names set to nil then change it to non-nil, the
> agent will not be able to find any of the content fetched before
> changing nnmail-use-long-file-names.

Ok. I understand this. But would it be difficult to change this behaviour or at
least add a function to update the agent ? I can start to write something but my
knowledge is quite limitted in Gnus (too many things). The trick would be to
force agent to re-read all its hierarchy and update it in consequence.

So renaming old 'foo/bar' to 'foo.bar' will not impact anything.

> > Not that performance only aplly for nntp groups since I recently
> > switched to nnimap.
> 
> I'll see if I can reproduce the problem here.

My impression is that Gnus tries to download its active (dunno if it is the
right term) file for the current group whenever I enter it. On small group it is
not a problem but entering gmane.emacs.gnus.general is really really long.
 
> > I have other problems with IMAP though.
>
> Sorry, you've already exceeded my knowledge of nnimap.

No problem :)

zeDek

Re: 'g' : get-new-news and fetch articles with agent and other

[Jørn Helge B. Dahl]

* Xavier Maillard <zedek@gnu-rox.org>

[...]

> My impression is that Gnus tries to download its active (dunno if it is the
> right term) file for the current group whenever I enter it. On small group it is
> not a problem but entering gmane.emacs.gnus.general is really really long.

Do you have gnus-agent-consider-all-articles set to t?  I fell
over that one.  I used it because I wanted to be completely
synced with my IMAP-server, but quickly I turned it off because
of the performance impact on big groups.

-- 
Jørn Helge B. Dahl

Re: 'g' : get-new-news and fetch articles with agent and other

[Kevin Greiner]

Xavier Maillard <zedek@gnu-rox.org> writes:

> Kevin Greiner <kgreiner <at> xpediantsolutions.com> writes:
>
>> > What do you mean by that ? 
>> 
>> The agent has no way to know that nnmail-use-long-file-names changed
>> value.  Therefore, if you use the agent with
>> nnmail-use-long-file-names set to nil then change it to non-nil, the
>> agent will not be able to find any of the content fetched before
>> changing nnmail-use-long-file-names.
>
> Ok. I understand this. But would it be difficult to change this behaviour or at
> least add a function to update the agent ? I can start to write something but my
> knowledge is quite limitted in Gnus (too many things). The trick would be to
> force agent to re-read all its hierarchy and update it in consequence.
>
> So renaming old 'foo/bar' to 'foo.bar' will not impact anything.

The agent could internally bind nnmail-use-long-file-names to a fixed
value (either nil or t would work) so that it is no longer sensitive
to a user changing nnmail-use-long-file-names.  It would probably be a
good deal of work and break the agent for all those users whose
personal nnmail-use-long-file-names had the opposite value.

A function is possible, even necessary if I did the previous.  It's
just that I'm already busy and I don't see that much demand for it.

If you want to write something to repair your own installation. You
should start by cloning gnus-agent-expire.  This function contains the
logic to iterate over all agentized groups.  The clone would do
something with gnus-agent-group-path to compute the old/new group path
names rather than call gnus-agent-expire-group-1.


>> > Not that performance only aplly for nntp groups since I recently
>> > switched to nnimap.
>> 
>> I'll see if I can reproduce the problem here.

I've now re-tested the agent with nnmail-use-long-file-names set to t
(I normally leave it set to nil).  The agent worked fine.  One
difference could be that I didn't try to repair my existing agent
hierarchy but instead renamed News/agent as News/agent.old.  The
result was that I had to completely start over using the agent.

> My impression is that Gnus tries to download its active (dunno if it is the
> right term) file for the current group whenever I enter it. On small group it is
> not a problem but entering gmane.emacs.gnus.general is really really long.
  
Changing nnmail-use-long-file-names would explain having to download
the entire active file ONCE.  Something else must be going on if gnus
is repeatedly downloading it.

If gnus is slow enough, you could set debug-on-quit then type cntl-g
while gnus if next fetching the active file.  The stacktrace will then
show which agent function(s) are involved in the decision to re-load
the entire active file.

Kevin

Re: 'g' : get-new-news and fetch articles with agent and other

[Xavier Maillard]

Jørn Helge B. Dahl <jornd <at> users.sourceforge.net> writes:

> Do you have gnus-agent-consider-all-articles set to t?  I fell

Yup I have this one set to t.

> over that one.  I used it because I wanted to be completely
> synced with my IMAP-server, but quickly I turned it off because
> of the performance impact on big groups.

Hmm and what do you do now to be completely synced ?? 

zeDek 

Re: 'g' : get-new-news and fetch articles with agent and other

[Xavier Maillard]

Kevin Greiner <kgreiner <at> xpediantsolutions.com> writes:

> The agent could internally bind nnmail-use-long-file-names to a fixed
> value (either nil or t would work) so that it is no longer sensitive
> to a user changing nnmail-use-long-file-names. 

Ok.

> just that I'm already busy and I don't see that much demand for it.

Yes it is not a common use to do what I did :)
 
> I've now re-tested the agent with nnmail-use-long-file-names set to t

 <snip>

> The result was that I had to completely start over using the agent.

Ok thanks for the feedback.
 
> Something else must be going on if gnus is repeatedly downloading it.

Yep I just had a clue on something causing this behaviour.
 
> If gnus is slow enough, you could set debug-on-quit then type cntl-g
> while gnus if next fetching the active file.

I will do this tonight anyway thank you for your help ! I think I will try 
to write a function to fix my current installation and maybe if it is of 
keen interest to anybody, I could put it online ;)

Regards,

zeDek

Re: 'g' : get-new-news and fetch articles with agent and other

[Jørn Helge B. Dahl]

* Xavier Maillard <zedek@gnu-rox.org>

> Jørn Helge B. Dahl <jornd <at> users.sourceforge.net> writes:
>
>> Do you have gnus-agent-consider-all-articles set to t?  I fell
>
> Yup I have this one set to t.
>
>> over that one.  I used it because I wanted to be completely
>> synced with my IMAP-server, but quickly I turned it off because
>> of the performance impact on big groups.
>
> Hmm and what do you do now to be completely synced ?? 

Well, I had it on, did a fetch, turned it off and then I never
agent-expire my IMAP groups. :-)

I let my nntp-groups expire once in a while.

-- 
Jørn Helge B. Dahl

Re: 'g' : get-new-news and fetch articles with agent and other

[Kai Grossjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> nnagent uses nnmail to read offline articles hence the dependency on
> nnmail-use-long-file-names.  Changing the value of
> nnmail-use-long-file-names is certainly a bad thing to do in that
> gnus-agent will not be able to find anything that it previously
> fetched.

Note that Gnus might change the value of nnmail-use-long-file-names
behind your back.  This causes grief.  The default value of
nnmail-use-long-file-names depends on the OS in use -- I was quite
surprised when moving between Solaris and GNU/Linux and FreeBSD that I
couldn't continue to use my home dir...

I suggest that Gnus look at startup whether filenames of the "wrong"
format are in use.  If so, barf.  If only I had time to implement
it... :-(

Kai

Re: 'g' : get-new-news and fetch articles with agent and other

[Barry Fishman]

Kai Grossjohann <kai@emptyhost.emptydomain.de> writes:
> Note that Gnus might change the value of nnmail-use-long-file-names
> behind your back.  This causes grief.  The default value of
> nnmail-use-long-file-names depends on the OS in use -- I was quite
> surprised when moving between Solaris and GNU/Linux and FreeBSD that I
> couldn't continue to use my home dir...
>
> I suggest that Gnus look at startup whether filenames of the "wrong"
> format are in use.  If so, barf.  If only I had time to implement
> it... :-(

Wouldn't be simpler to just use `nnmail-use-long-file-name' when
setting up the agent hierarchy for the first time.  Afterwards it
would just quietly use the hierarchy as it was created.  If necessary
a `state' file at the top of the hierarchy could be used to identify
its structure quickly.

Changing the agent hierarchy format could then later be added as a
function(s) which could be run a time chosen by the user.