Regenerate .agentview files?

Regenerate .agentview files?

[Kai Großjohann]

I have a group which has a lot of superfluous entries in .agentview:
there are downloaded articles 30208 through 30969 (and undownloaded
articles 30970 and 30971), but .agentview contains entries (112032)
through (112897)...

I've looked at the code but I'm not sure if gnus-agent-regenerate
uses .agentview as input or creates a new one as output.

The .overview seems to contain the correct headers (ie, 30208 through
30969), FWIW.

[...time passes...]

Ah.  I've now tried the command, and it regenerated agent.lib/active
files, but the .agentview hasn't changed.  Hm.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Regenerate .agentview files?

[Kai Großjohann]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> I have a group which has a lot of superfluous entries in .agentview:
> there are downloaded articles 30208 through 30969 (and undownloaded
> articles 30970 and 30971), but .agentview contains entries (112032)
> through (112897)...

I've now concocted the following abomination which appears to have
been effective, if not efficient.

(defun kai-gnus-agent-iterate-groups (function)
  "Iterate over all available groups and call FUNCTION for each."
  (let ((methods gnus-agent-covered-methods)
	groups group gnus-command-method)
    (while methods
      (message "doing method %s" (car methods))
      (setq gnus-command-method (car methods))
      (setq groups (gnus-groups-from-server (car methods)))
      (gnus-agent-with-fetch
	(while (setq group (pop groups))
	  (when (<= (gnus-group-level group) gnus-agent-handle-level)
	    (funcall function group))))
      (pop methods))))

(defun kai-gnus-agent-prune-agentview (group)
  (message "group: %s" group)
  (gnus-agent-load-alist group)
  (let ((articles (copy-sequence gnus-agent-article-alist))
	art)
    (while articles
      (setq art (pop articles))
      (unless (file-exists-p (gnus-agent-article-name
			      (format "%s" (car art))
			      group))
	(setq gnus-agent-article-alist (delete art gnus-agent-article-alist))))
    (gnus-agent-save-alist group))
  (message "group: %s...done" group))

(kai-gnus-agent-iterate-groups 'kai-gnus-agent-prune-agentview)

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Regenerate .agentview files?

[Steinar Bang]

>>>>> kai.grossjohann@uni-duisburg.de (Kai Großjohann):

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>> I have a group which has a lot of superfluous entries in .agentview:
>> there are downloaded articles 30208 through 30969 (and undownloaded
>> articles 30970 and 30971), but .agentview contains entries (112032)
>> through (112897)...

> I've now concocted the following abomination which appears to have
> been effective, if not efficient.

Will it regenerate the NOV cache of agentized nntp groups?  (I would
like reported unspam articles to reappear in the groups)

Re: Regenerate .agentview files?

[Kai Großjohann]

Steinar Bang <sb@dod.no> writes:

> Will it regenerate the NOV cache of agentized nntp groups?  (I would
> like reported unspam articles to reappear in the groups)

No, it just removes things from .agentview.

But it would be desirable to have code which reads the downloaded
articles and builds a .overview file containing them.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Regenerate .agentview files?

[Steinar Bang]

>>>>> kai.grossjohann@uni-duisburg.de (Kai Großjohann):

> Steinar Bang <sb@dod.no> writes:

>> Will it regenerate the NOV cache of agentized nntp groups?  (I
>> would like reported unspam articles to reappear in the groups)

> No, it just removes things from .agentview.

> But it would be desirable to have code which reads the downloaded
> articles and builds a .overview file containing them.

Does that mean there isn't already a cached NOV file in an agentized
nntp group?  Or do you mean "rebuild the existing .overview" file?

Re: Regenerate .agentview files?

[Kai Großjohann]

Steinar Bang <sb@dod.no> writes:

>>>>>> kai.grossjohann@uni-duisburg.de (Kai Großjohann):
>
>> But it would be desirable to have code which reads the downloaded
>> articles and builds a .overview file containing them.
>
> Does that mean there isn't already a cached NOV file in an agentized
> nntp group?  Or do you mean "rebuild the existing .overview" file?

I sometimes get hosed .overview files in ~/News/agent.  It would be
nice if it was possible to delete them and then to tell Gnus to go
through the articles already downloaded and to make a small .overview
file which contains data about those articles.  The next time I enter
the group the .overview file will then contain more data fetched from
the server, I guess.

It would just be a help for developing Gnus, I don't expect that
normal users will use it a lot.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Regenerate .agentview files?

[Steinar Bang]

>>>>> kai.grossjohann@uni-duisburg.de (Kai Großjohann):

> It would just be a help for developing Gnus, I don't expect that
> normal users will use it a lot.

If it can be used to update agent overviews for gmane.org groups, I
would use it quite a bit.

Re: Regenerate .agentview files?

[Danny Siu]

It is very useful to be able to regenerate .agentview file from existing
downloaded articles cuz in my case, I have nearly 400MB emails on my IMAP
server and it would be a time saver without having to redownloading all when
agent behaviour get changed.

Kai Großjohann writes:

  Kai> I sometimes get hosed .overview files in ~/News/agent.  It would be
  Kai> nice if it was possible to delete them and then to tell Gnus to go
  Kai> through the articles already downloaded and to make a small .overview
  Kai> file which contains data about those articles.  The next time I enter
  Kai> the group the .overview file will then contain more data fetched from
  Kai> the server, I guess.

  Kai> It would just be a help for developing Gnus, I don't expect that
  Kai> normal users will use it a lot.

-- 
Danny Siu

Re: Regenerate .agentview files?

[Kai Großjohann]

Danny Siu <dsiu@adobe.com> writes:

> It is very useful to be able to regenerate .agentview file from existing
> downloaded articles cuz in my case, I have nearly 400MB emails on my IMAP
> server and it would be a time saver without having to redownloading all when
> agent behaviour get changed.

Yes, the agent should definitely look at the disk from time to time,
and if there is a file in the directory, it should add corresponding
entries in .overview and .agentview.

Hm.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Regenerate .agentview files?

[Steinar Bang]

>>>>> kai.grossjohann@uni-duisburg.de (Kai Großjohann):

> Danny Siu <dsiu@adobe.com> writes:

>> It is very useful to be able to regenerate .agentview file from
>> existing downloaded articles cuz in my case, I have nearly 400MB
>> emails on my IMAP server and it would be a time saver without
>> having to redownloading all when agent behaviour get changed.

> Yes, the agent should definitely look at the disk from time to time,
> and if there is a file in the directory, it should add corresponding
> entries in .overview and .agentview.

Just making a note that my wish is different: I want the agent
.overview to be regenerated when the NOV on the NNTP server have been
changed. 

Changing the NOV does not happen on normal NNTP servers, according to
larsi.  The only one where I know it happens, is news.gmane.org, when
articles mistakenly marked as spam (by a crosspost to a detected spam
group), has been manually unspammed (ie. the crosspost has been
removed), and vice versa, spam missed by SpamAssassin has been
manually crossposted to the detected spam group.

Re: Regenerate .agentview files?

[Kai Großjohann]

Steinar Bang <sb@dod.no> writes:

> Just making a note that my wish is different: I want the agent
> .overview to be regenerated when the NOV on the NNTP server have been
> changed. 

How can this be done?  The point of the local .overview file is that
you don't have to retrieve the NOV data.  And if you don't retrieve
the NOV data from the server, how can you know it has changed?

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Regenerate .agentview files?

[Steinar Bang]

>>>>> kai.grossjohann@uni-duisburg.de (Kai Großjohann):

> Steinar Bang <sb@dod.no> writes:

>> Just making a note that my wish is different: I want the agent
>> .overview to be regenerated when the NOV on the NNTP server have
>> been changed.

> How can this be done?  The point of the local .overview file is that
> you don't have to retrieve the NOV data.  And if you don't retrieve
> the NOV data from the server, how can you know it has changed?

I want to trigger the download manually when I know that a change to
the server's NOV has been done (typically an unspam in gmane), so that
the article is no longer scored down by my spam removal rule.