Prevent B c/B m from createing dups in target group

Prevent B c/B m from createing dups in target group

[Harry Putnam]

[ ALERT Repost from gnu.emacs.gnus - possible windbag length ]

There are those of us who do a fair amount of copying of messages from
one group to another.  Maybe editing beforehand or maybe not.  I keep
quite a few messages that I have added keywords header to in such
groups as nnml:t.security, or nnml:t.networking.  These groups catch
copies of posts or mail I thought to be particularly pertinent to the
subject and wanted to save for later reference.  Some times I edit them
sometimes not.

Some are whole threads that were copied a piece at a time as the
thread was developing.  Inevitabley there comes a point where you've
forgotten what has been copied and what not.  I found as many as four
layers of copies in some groups.  Partly due to absentmindedness but
also there is no built in functionality in gnus to prevent this.  Near
as I can tell.

I've often thought to try to figure out how to make gnus compare a
message ID or group of messages ids to those already in the group.
Most often (in my case) this would not be large numbers.  Probably
below 200 in nearly every case.

Some kind of hook to run when `B c' or `B m' is being used that would
initiate a function that took the message ID of message under point or
messages marked with # and compared each message id to those in the group.

I'd like to see a message that tells me some of these are duplicates and
offers to move only the unique ones or to go ahead with the dups as I wish.

Remberering from 4-5 years ago with gnus-agents' name sake Forte agent,
this was a feature in Forte agent.

Being generally lisp impaired, what I'm asking for here is general
outline of things to look into to get an understanding of what has to
happen to create this kind of functionality.

What functions to hook into.  In general terms, how to get started
coding something like this.

I can image writing such stuff in awk, not even very difficult in
plain shell scripting with korn/bash arrays and for loops.  But have
no clue how to pass the message-ids to such scripting and imagine it
would be smarter to use lisp functions for the tasks all the way
through.  Maybe no more complicated than telling gnus to keep a
Message-ID cache on per group basis, then code a comparison routine
that gets called by the user or is hooked into `B c', `B m'.

Any tips to get started or places to start po[ou]ring(?) over the code will
be appreciated.


 

Re: Prevent B c/B m from createing dups in target group

[Kai Großjohann]

I wonder if one of the variables help that are about duplicates in
Gnus?

M-x apropos-variable RET gnus.*dup RET

I haven't tried it, but it's worth a shot.  Little effort for some
potential gain.

kai
-- 
~/.signature: No such file or directory

Re: Prevent B c/B m from createing dups in target group

[Harry Putnam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> I wonder if one of the variables help that are about duplicates in
> Gnus?
> 
> M-x apropos-variable RET gnus.*dup RET
> 

Yeah, I knew about those but the first relies on splitting or
respooling which is not activated in B c or B m usage.

And it seems that at least one of the others depends on that `Warning'
thing being present.  Even if the middle one (below) checks message
IDs it still only hides there presence by my reading.  So that other
ways of accessing that directory would still find them.

Further, none of it has any provision for notifying the user that they
are dealing with dups and allowing a choice of proceeding or not.
( That is, other than the 'Warning' header in the case of split stuff )

With that warning mechanism in place, and provision to copy/move only
the unique messages on command, a user could simply throw a batch of
messages at a directory and only the unique stuff would stick...

========================================

gnus-duplicate-mark
  Variable: *Mark used for duplicate articles.
gnus-summary-ignore-duplicates
  Variable: *If non-nil, ignore articles with identical Message-ID headers.
gnus-suppress-duplicates
  Variable: *If non-nil, Gnus will mark duplicate copies of the same article as read.

Re: Prevent B c/B m from createing dups in target group

[Lars Magne Ingebrigtsen]

Harry Putnam <reader@newsguy.com> writes:

> With that warning mechanism in place, and provision to copy/move only
> the unique messages on command, a user could simply throw a batch of
> messages at a directory and only the unique stuff would stick...

Yeah.  But there is no predefined functionality in Gnus for doing
this, and I think it sounds like something of a non-problem.  :-)
When you suppress duplicate messages on group entry, you usually just
see one copy in any case, no matter how many are actually there...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Prevent B c/B m from createing dups in target group

[Paul Jarc]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> When you suppress duplicate messages on group entry, you usually just
> see one copy in any case, no matter how many are actually there...

But they're still visible from outside Gnus, and they still take up
space.


paul

Re: Prevent B c/B m from createing dups in target group

[Lars Magne Ingebrigtsen]

prj@po.cwru.edu (Paul Jarc) writes:

> But they're still visible from outside Gnus, and they still take up
> space.

Outside Gnus?  Is there such a place?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Prevent B c/B m from createing dups in target group

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yeah.  But there is no predefined functionality in Gnus for doing
> this, and I think it sounds like something of a non-problem.  :-)
> When you suppress duplicate messages on group entry, you usually just
> see one copy in any case, no matter how many are actually there...

Well, but Harry is copying messages, and copying the same message
several times doesn't make sense.

The reason Harry is copying several times is because he sees an
interesting thread, then possibly edits the messages in that thread,
then he copies the messages into an archive group.  There's no way for
him to (easily) see which messages were copied already.

Hm.  Maybe a different approach is possible altogether?

If it wasn't for the editing, saving the messages in the cache (via
`*') would be enough.  Hm.

kai
-- 
~/.signature: No such file or directory

Re: Prevent B c/B m from createing dups in target group

[Kai Großjohann]

Harry Putnam <reader@newsguy.com> writes:

> Some are whole threads that were copied a piece at a time as the
> thread was developing.  Inevitabley there comes a point where you've
> forgotten what has been copied and what not.  I found as many as four
> layers of copies in some groups.  Partly due to absentmindedness but
> also there is no built in functionality in gnus to prevent this.  Near
> as I can tell.

You could also (setq gnus-use-cache t) and then tick the interesting
messages.  Is there a specific reason for you not to do that?

You say that you're editing messages.  What kind of editing is that?
Surely, just putting them in the cache wouldn't be enough when you
want to edit them.

But OTOH, why not allow editing of articles which are in the cache?
That might be a useful feature.

kai
-- 
~/.signature: No such file or directory

Re: Prevent B c/B m from createing dups in target group

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Well, but Harry is copying messages, and copying the same message
> several times doesn't make sense.

Maybe not, but on the other hand -- does it really matter?

> If it wasn't for the editing, saving the messages in the cache (via
> `*') would be enough.  Hm.

But if you could edit the articles in the cache...  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Prevent B c/B m from createing dups in target group

[Harry Putnam]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Harry Putnam <reader@newsguy.com> writes:
> 
> > With that warning mechanism in place, and provision to copy/move only
> > the unique messages on command, a user could simply throw a batch of
> > messages at a directory and only the unique stuff would stick...
> 
> Yeah.  But there is no predefined functionality in Gnus for doing
> this..

My OP was asking for an outline of how to go about starting to create
that functionality in gnus.  Once finished it could hopefully be
submitted for consideration as an inclusion.

I'm pretty sure once this kind of functionality became known it would
be seen as a nice addition.

>  and I think it sounds like something of a non-problem.  :-)

> When you suppress duplicate messages on group entry, you usually just
> see one copy in any case, no matter how many are actually there...

Possibly it is a non-problem but I see at least a few problems  with
that approach:

1) In the event I have edited the first copy in some way.  Then later
   looking back at old threads I see that same message not realizing
   I've already copied and edited it to a save group.  I copy it again
   but being in a hurry neglect to edit it.  Which one will show up
   when duplicate suppression on entry is engaged?

2) I run up on an old thread in a Newsgroup.  Maybe a discussion of
   `sender' and RFC provisions and decide I'd like to keep the whole
   thing...  all 235 : ) messages.  I've already piece meal saved 70,
   not necessaryily in order.  I just throw the whole batch into a
   saved group, now I have at least 70 dups.  Do I want that overhead?
   When I might possible be able to write something for gnus that
   would allow me to do that and only actually move the unique ones on
   my request.

3) Suppression seems very much like taking pain pills to mask an
   injury while continuing to grind your body to dust...

Re: Prevent B c/B m from createing dups in target group

[Lars Magne Ingebrigtsen]

Harry Putnam <reader@newsguy.com> writes:

> My OP was asking for an outline of how to go about starting to create
> that functionality in gnus.  Once finished it could hopefully be
> submitted for consideration as an inclusion.

Oh, right.

In that case, here's how I would do it.

I'd make an option to have mail backends cache the group/message-id
pair somewhere.  Take the nnmail.*cache.* functions as a starting
point.  (They only cache on a message-id basis.)  Then have an option
to refuse entry into the mail backends when you find a
group/message-id pair that matches.

The problem with this approach is that if you move a message out of a
group, you have to remove it from the cache as well.  And what happens
if you delete it outside of Gnus?  You wouldn't then be able to copy a
new copy into the group.

Scratch that.

Perhaps the nn.*-request-accept-article function could have an option
to go through all the messages in the group and see whether the
message is already there.  But that would take a looong time.

> 3) Suppression seems very much like taking pain pills to mask an
>    injury while continuing to grind your body to dust...

On the other hand, if you have a slight headache, a pain killer makes
it all go away and makes you happy and shiny again.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Prevent B c/B m from createing dups in target group

[Harry Putnam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Yeah.  But there is no predefined functionality in Gnus for doing
> > this, and I think it sounds like something of a non-problem.  :-)
> > When you suppress duplicate messages on group entry, you usually just
> > see one copy in any case, no matter how many are actually there...
> 
> Well, but Harry is copying messages, and copying the same message
> several times doesn't make sense.
> 
> The reason Harry is copying several times is because he sees an
> interesting thread, then possibly edits the messages in that thread,
> then he copies the messages into an archive group.  There's no way for
> him to (easily) see which messages were copied already.
> 
> Hm.  Maybe a different approach is possible altogether?
> 
> If it wasn't for the editing, saving the messages in the cache (via
> `*') would be enough.  Hm.

Hmm maybe, but would that still allow a grouping under an appropriate
named group.  That is, would I then be able to display a possibly
cross group collecton in a single buffer.  I think not, but not really
sure of all possibilites with cache.  I haven't used it since gnus
went agent.

Maybe something like this would serve the purpose:

cache special messages in there respective groups then run nnir on the
cache against the subject or other reference. Possibly an added
keywords header.

Hey, that sounds like it might be worthwhile.  Limiting nnir home to
just the cache.  I don't use glimpse or wais anymore on the full mail
news collection.  But it would be fast as a greased bullit against
just the cache.

Re: Prevent B c/B m from createing dups in target group

[Paul Jarc]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> Perhaps the nn.*-request-accept-article function could have an option
> to go through all the messages in the group and see whether the
> message is already there.  But that would take a looong time.

That depends on the backend.  nnmaildir could check this quickly,
because it keeps an obarray in memory for each group, where the symbol
names are the Message-IDs of the articles in the group.


paul

Re: Prevent B c/B m from createing dups in target group

[Harry Putnam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Harry Putnam <reader@newsguy.com> writes:
> 
> > Some are whole threads that were copied a piece at a time as the
> > thread was developing.  Inevitabley there comes a point where you've
> > forgotten what has been copied and what not.  I found as many as four
> > layers of copies in some groups.  Partly due to absentmindedness but
> > also there is no built in functionality in gnus to prevent this.  Near
> > as I can tell.
> 
> You could also (setq gnus-use-cache t) and then tick the interesting
> messages.  Is there a specific reason for you not to do that?

Only as mentioned in previous post (not on server when you wrote
above) that it is handy to have a collection of messages under a topic
named group.  Far as I remember cacheing a message leaves it in its
original group, is that correct?  Or cacheing from NNTP groups, do
those go to a single directory?

> You say that you're editing messages.  What kind of editing is that?
> Surely, just putting them in the cache wouldn't be enough when you
> want to edit them.

Most often very light editing or none.  By light I mean adding a
keyword or two to Subject line or adding an entire new Keywords header.

Rarely a full body edit, but sometimes.

> But OTOH, why not allow editing of articles which are in the cache?
> That might be a useful feature.

Yeah and in conjuction with nnir could be used to create the cross
group summary buffer I'd like to have.  Assuming nnir can be aimed at
the cache.

Re: Prevent B c/B m from createing dups in target group

[Lars Magne Ingebrigtsen]

prj@po.cwru.edu (Paul Jarc) writes:

> That depends on the backend.  nnmaildir could check this quickly,
> because it keeps an obarray in memory for each group, where the symbol
> names are the Message-IDs of the articles in the group.

Yup.  And it could be pretty fast in nnml groups as well -- it has NOV
files.  It would be really slow in nnmh groups.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Prevent B c/B m from createing dups in target group

[Harry Putnam]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

[...]
Thanks for the quick survey

> Perhaps the nn.*-request-accept-article function could have an option
> to go through all the messages in the group and see whether the
> message is already there.  But that would take a looong time.

Now we're talking.  From an absolute amateur viewpoint, I'm thinking would this
really be so long, unless the group was rilly rilly big.  It takes only
10 seconds or so for gnus to generate a full nov file in a group of  1200
and it seems that would be much more work than comparing say 5
different IDs, one at a time, againt say 1000.  The comparison would
run at most 5,000 times, maybe to much .. is it? or maybe some way to
speed it up.

> > 3) Suppression seems very much like taking pain pills to mask an
> >    injury while continuing to grind your body to dust...
> 
> On the other hand, if you have a slight headache, a pain killer makes
> it all go away and makes you happy and shiny again.

He he.. yeah but for that comparison to really stand up, it would have
to be added that sometimes just taking off or putting on your glasses
may remove the headache.  I guess I'm talking about this code being
comprable to that little chore and probably it isn't eh?

Re: Prevent B c/B m from createing dups in target group

[Kai Großjohann]

Harry Putnam <reader@newsguy.com> writes:

> Yeah and in conjuction with nnir could be used to create the cross
> group summary buffer I'd like to have.  Assuming nnir can be aimed at
> the cache.

In principle, it sure can.  As long as it gets (group name, article
number) pairs, it's happy.  Doesn't matter whether those come from the
cache or not.

The urge is getting stronger and stronger to provide multiple backends
capability in nnir.el (in the sense of querying several search engines
at once), but it's been on the agenda for far too long now.  I don't
have too much hope :-(  So if anybody wants to hack it, please do so.

kai
-- 
~/.signature: No such file or directory

Re: Prevent B c/B m from createing dups in target group

[Nuutti Kotivuori]

Kai Großjohann wrote:
> But OTOH, why not allow editing of articles which are in the cache?
> That might be a useful feature.

Argh, patch is almost finished for this (has been for a few weeks, but
there were too many bugs in cache handling to submit it then), will be
submitted when I finally have a free couple hours.

-- Naked