Concerning marks and the back end.

Concerning marks and the back end.

[Lloyd Zusman]

Since gnus is making use of `.marks' files these days, I'm wondering if
there's a way to solve the following problem that I'm having:

In some groups, I keep a large history of old messages.  Therefore, it
can be very slow to enter such a group.  If I want to speed things up,
one thing that I know I could do would be to create some sort of
auxiliary group for each of these groups, wherein I can put most of the
old articles.  This way, under normal, everyday use, I can enter the
main group relatively quickly, and those more infrequent times when I
want to look through the old articles, I can go to the auxiliary group.

But I'm wondering if I can avoid these auxiliary groups.  I'm wondering
if there's a mark or set of marks that I can put on articles, and whose
existence would be noted in the .marks file for any given group.  And if
so, would it be possible to set up a group so that under normal group
entry, any article ranges that are marked with this special mark will
cause the articles to be ignored with no query being done to the back
end? ... and only if I enter a group in a non-standard way (i.e., via
a different command or a special prefix argument) would these special
marks be ignored and the entire list of articles would be quieried
from the back end.

This way, I could keep all articles in their original group, with the
oldest articles marked in this special way.  Entering the group in the
"normal" manner would be fast, since no back-end query would need to be
done for any articles for which the .marks file indicates the existence
of this special mark.  These back-end queries would only be performed
for the non-marked (in this case, newest) articles.

I hope I explained this clearly.

Is this something that can now be done in some way, or which perhaps
could be added without a huge amount of complexity?

Thanks in advance.

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Kai Großjohann]

Lloyd Zusman <ljz@asfast.com> writes:

> In some groups, I keep a large history of old messages.  Therefore, it
> can be very slow to enter such a group.

What's slow?  I think fetching headers is not too bad, so if you just
mark the old messages as read, that should do the trick.
-- 
Ambibibentists unite!

Re: Concerning marks and the back end.

[Lloyd Zusman]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> In some groups, I keep a large history of old messages.  Therefore, it
>> can be very slow to enter such a group.
>
> What's slow?  I think fetching headers is not too bad, so if you just
> mark the old messages as read, that should do the trick.

I'm talking about the case where all messages are already marked as
read, and I want to enter the group.  If there is a several-thousand
message backlog of already-read messages in that group, it takes a huge
amount of time to construct the summary buffer.  For example, one such
group has 5670 old messages as of this posting.

Not fetching headers for most of this several-thousand message backlog
would speed things up immensely.


> -- 
> Ambibibentists unite!
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Lars Magne Ingebrigtsen]

Lloyd Zusman <ljz@asfast.com> writes:

> I'm talking about the case where all messages are already marked as
> read, and I want to enter the group.  If there is a several-thousand
> message backlog of already-read messages in that group, it takes a huge
> amount of time to construct the summary buffer.  For example, one such
> group has 5670 old messages as of this posting.
>
> Not fetching headers for most of this several-thousand message backlog
> would speed things up immensely.

If you only ask for the, say, three last articles, you should only
get the article headers for (approx.) the last three articles.

Or is it the .marks code that goes through a lot more articles than
that?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Concerning marks and the back end.

[Simon Josefsson]

Lloyd Zusman <ljz@asfast.com> writes:

> Since gnus is making use of `.marks' files these days, I'm wondering if
> there's a way to solve the following problem that I'm having:
>
> In some groups, I keep a large history of old messages.  Therefore, it
> can be very slow to enter such a group.  If I want to speed things up,
> one thing that I know I could do would be to create some sort of
> auxiliary group for each of these groups, wherein I can put most of the
> old articles.  This way, under normal, everyday use, I can enter the
> main group relatively quickly, and those more infrequent times when I
> want to look through the old articles, I can go to the auxiliary group.
>
> But I'm wondering if I can avoid these auxiliary groups.  I'm wondering
> if there's a mark or set of marks that I can put on articles, and whose
> existence would be noted in the .marks file for any given group.  And if
> so, would it be possible to set up a group so that under normal group
> entry, any article ranges that are marked with this special mark will
> cause the articles to be ignored with no query being done to the back
> end? ... and only if I enter a group in a non-standard way (i.e., via
> a different command or a special prefix argument) would these special
> marks be ignored and the entire list of articles would be quieried
> from the back end.
>
> This way, I could keep all articles in their original group, with the
> oldest articles marked in this special way.  Entering the group in the
> "normal" manner would be fast, since no back-end query would need to be
> done for any articles for which the .marks file indicates the existence
> of this special mark.  These back-end queries would only be performed
> for the non-marked (in this case, newest) articles.
>
> I hope I explained this clearly.
>
> Is this something that can now be done in some way, or which perhaps
> could be added without a huge amount of complexity?

nnvirtual could solve this.

But the real problem seem to be why it is taking too long.  Maybe that
part can be optimized away, e.g. by not querying for articles Gnus can
figure out will never be shown in the summary buffer anyway.  I added
the following to the Troubleshooting chapter, does it help in
isolating the slow part?  If it is incorrect in any way (written from
memory, not tested, etc) please point that out too.  Thanks.

,----
|    Sometimes, a problem do not directly generate a elisp error but
| manifests itself by causing Gnus to be very slow.  In these cases, you
| can use `M-x toggle-debug-on-quit' and press C-j when things are slow,
| and then try to analyze the backtrace (repeating the procedure helps
| isolating the real problem areas).  A fancier approach is to use the
| elisp profiler, ELP.  The profiler is (or should be) fully documented
| elsewhere, but to get you started there are a few steps that need to be
| followed.  First, instrument the part of Gnus you are interested in for
| profiling, e.g. `M-x elp-instrument-package RET gnus' or `M-x
| elp-instrument-packagre RET message'.  Then perform the operation that
| is slow and press `M-x elp-results'.  You will then see which
| operations that takes time, and can debug them further.  If the entire
| operation takes much longer than the time spent in the slowest function
| in the profiler output, you probably profiled the wrong part of Gnus.
| To reset profiling statistics, use `M-x elp-reset-all'.  `M-x
| elp-restore-all' is supposed to remove profiling, but given the
| complexities and dynamic code generation in Gnus, it might not always
| work perfectly.
`----

Re: Concerning marks and the back end.

[Lloyd Zusman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> I'm talking about the case where all messages are already marked as
>> read, and I want to enter the group.  If there is a several-thousand
>> message backlog of already-read messages in that group, it takes a huge
>> amount of time to construct the summary buffer.  For example, one such
>> group has 5670 old messages as of this posting.
>>
>> Not fetching headers for most of this several-thousand message backlog
>> would speed things up immensely.
>
> If you only ask for the, say, three last articles, you should only
> get the article headers for (approx.) the last three articles.

Well, I guess you're talking about my typing something like ...

  <ESC> 5 0 <space>

... when pointing to the group in the Groups buffer to see only that
last (approx.) 50 unread articles.  I had forgotten about that and it
indeed works for me.  So for the time being, this is a big help.

Nonetheless, I'm wondering if there's a way to mark articles in such a
way that these marks are stored in the .marks file (instead of the back
end), so that articles marked in this way will never be queried from the
back end unless some special command is typed.

It seems like that would be a nice feature.

Well, maybe in my not-very-voluminous-yet-more-than-nonexistent spare
time I'll look at the source and see if I could hack^H^H^H^Hcode up
something like that.


> Or is it the .marks code that goes through a lot more articles than
> that?

I'm not sure what you're asking, but I think the answer is no :) ...
given that the above test worked for me.


> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    larsi@gnus.org * Lars Magne Ingebrigtsen
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Lloyd Zusman]

Simon Josefsson <jas@extundo.com> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> [ ... etc. ... ]
>
> nnvirtual could solve this.
>
> But the real problem seem to be why it is taking too long.  Maybe that
> part can be optimized away, e.g. by not querying for articles Gnus can
> figure out will never be shown in the summary buffer anyway.  I added
> the following to the Troubleshooting chapter, does it help in
> isolating the slow part?  If it is incorrect in any way (written from
> memory, not tested, etc) please point that out too.  Thanks.


I'll do some profiling over the next day or two and send the results.
I'm happy to do so.  Stay tuned ...

And thanks to you, too.


> ,----
> |    [ ... more etc. ... ]
> `----
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Simon Josefsson]

Lloyd Zusman <ljz@asfast.com> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Lloyd Zusman <ljz@asfast.com> writes:
>>
>>> I'm talking about the case where all messages are already marked as
>>> read, and I want to enter the group.  If there is a several-thousand
>>> message backlog of already-read messages in that group, it takes a huge
>>> amount of time to construct the summary buffer.  For example, one such
>>> group has 5670 old messages as of this posting.
>>>
>>> Not fetching headers for most of this several-thousand message backlog
>>> would speed things up immensely.
>>
>> If you only ask for the, say, three last articles, you should only
>> get the article headers for (approx.) the last three articles.
>
> Well, I guess you're talking about my typing something like ...
>
>   <ESC> 5 0 <space>
>
> ... when pointing to the group in the Groups buffer to see only that
> last (approx.) 50 unread articles.  I had forgotten about that and it
> indeed works for me.  So for the time being, this is a big help.
>
> Nonetheless, I'm wondering if there's a way to mark articles in such a
> way that these marks are stored in the .marks file (instead of the back
> end), so that articles marked in this way will never be queried from the
> back end unless some special command is typed.
>
> It seems like that would be a nice feature.

Maybe you can use this approach to accomplish something similar: set
the group parameter "display" (G c) to a value that makes group
entering fast (e.g. 500).  If you ever want to see older articles,
enter the group with C-u SPC.  If you want to keep around a specific
article even after falls outside of the 500 limit, press ! on it.

Re: Concerning marks and the back end.

[Lloyd Zusman]

Simon Josefsson <jas@extundo.com> writes:

> [ ... ]
>
> Maybe you can use this approach to accomplish something similar: set
> the group parameter "display" (G c) to a value that makes group
> entering fast (e.g. 500).  If you ever want to see older articles,
> enter the group with C-u SPC.  If you want to keep around a specific
> article even after falls outside of the 500 limit, press ! on it.

Aha!  Yes, that would work splendidly to give me pretty much exactly
what I want.

Great idea ... and now, as far as I'm concerned, Problem Solved.

Thanks a lot.

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Lloyd Zusman]

Lloyd Zusman <ljz@asfast.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> [ ... ]
>>
>> Maybe you can use this approach to accomplish something similar: set
>> the group parameter "display" (G c) to a value that makes group
>> entering fast (e.g. 500).  If you ever want to see older articles,
>> enter the group with C-u SPC.  If you want to keep around a specific
>> article even after falls outside of the 500 limit, press ! on it.
>
> Aha!  Yes, that would work splendidly to give me pretty much exactly
> what I want.
>
> Great idea ... and now, as far as I'm concerned, Problem Solved.
>
> Thanks a lot.

... but then again ...

I just noticed something less-than-ideal about this.  For example,
suppose a group has 10 unread messages and 6 zillion already read
messages.  I set the "display" paramter to 100 as discussed above, and I
then enter the group.  I now see the 10 new unread messages interspersed
among 90 already-read messages.

Is there any way to make the "display" parameter only apply when there
are no unread messages at all in the group?

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Simon Josefsson]

Lloyd Zusman <ljz@asfast.com> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> [ ... ]
>>>
>>> Maybe you can use this approach to accomplish something similar: set
>>> the group parameter "display" (G c) to a value that makes group
>>> entering fast (e.g. 500).  If you ever want to see older articles,
>>> enter the group with C-u SPC.  If you want to keep around a specific
>>> article even after falls outside of the 500 limit, press ! on it.
>>
>> Aha!  Yes, that would work splendidly to give me pretty much exactly
>> what I want.
>>
>> Great idea ... and now, as far as I'm concerned, Problem Solved.
>>
>> Thanks a lot.
>
> ... but then again ...
>
> I just noticed something less-than-ideal about this.  For example,
> suppose a group has 10 unread messages and 6 zillion already read
> messages.  I set the "display" paramter to 100 as discussed above, and I
> then enter the group.  I now see the 10 new unread messages interspersed
> among 90 already-read messages.
>
> Is there any way to make the "display" parameter only apply when there
> are no unread messages at all in the group?

You can set the display parameter to an array of predicates, e.g.:

[or unread (and no-unreads my-article-old-p)]

     (defun my-article-old-p ()
       "Say whether an article is old."
       (< (time-to-days (date-to-time (mail-header-date gnus-headers)))
          (- (time-to-days (current-time)) gnus-agent-expire-days)))

implementing the no-unreads function is left as an exercise. :-)

But this might not work either, as I think display predicates was
implemented as limiting, so you will fetch all article headers anyway.

But I still wonder why entering the group is slow for you.  I have
15,000 messages in my ding mailing list mailbox, which is stored on a
IMAP server, and entering it and viewing the unread and ticked
articles takes no time.  When I get several hundred thousand articles
it becomes a little slow though (long coffe break).

Re: Concerning marks and the back end.

[Lloyd Zusman]

Simon Josefsson <jas@extundo.com> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>>
>> [ ... ]
>>
>> Is there any way to make the "display" parameter only apply when there
>> are no unread messages at all in the group?
>
> You can set the display parameter to an array of predicates, e.g.:
>
> [or unread (and no-unreads my-article-old-p)]
>
>      (defun my-article-old-p ()
>        "Say whether an article is old."
>        (< (time-to-days (date-to-time (mail-header-date gnus-headers)))
>           (- (time-to-days (current-time)) gnus-agent-expire-days)))
>
> implementing the no-unreads function is left as an exercise. :-)
>
> But this might not work either, as I think display predicates was
> implemented as limiting, so you will fetch all article headers anyway.

Hmmm ... well, I could always cheat and use `defadvice' to wrap the
function which enters the group; and I would use `let' to cover the
variable that contains the `display' value that we discussed earlier.
The `no-unreads' value has got to be available somewhere in the Group
buffer, because it gets displayed to the left of the group name in that
buffer.  So if that no-unreads value is non-zero, I would set the
covered `display' value to nil, otherwise, to 100 or something.

Sounds easy, anyway. :)


> But I still wonder why entering the group is slow for you.  I have
> 15,000 messages in my ding mailing list mailbox, which is stored on a
> IMAP server, and entering it and viewing the unread and ticked
> articles takes no time.  When I get several hundred thousand articles
> it becomes a little slow though (long coffe break).

Well, I still will do the profiling that you suggested earlier.

I wonder if in my case things would be slower than for you because ...

1. I'm having this problem in nnml and nnspool groups, and not nnimap,
   which I'm not using these days.

2. I have a custom summary line format string that makes use of a
   couple of custom format functions that I wrote ... this will
   probably turn out to be the culprit.

Well, all will become much clearer after I run the profiler.

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Simon Josefsson]

Lloyd Zusman <ljz@asfast.com> writes:

>> But I still wonder why entering the group is slow for you.  I have
>> 15,000 messages in my ding mailing list mailbox, which is stored on a
>> IMAP server, and entering it and viewing the unread and ticked
>> articles takes no time.  When I get several hundred thousand articles
>> it becomes a little slow though (long coffe break).
>
> Well, I still will do the profiling that you suggested earlier.
>
> I wonder if in my case things would be slower than for you because ...
>
> 1. I'm having this problem in nnml and nnspool groups, and not nnimap,
>    which I'm not using these days.

Nnimap is rather inefficient and should be slower than nnml/nnspool on
local filesystems.

> 2. I have a custom summary line format string that makes use of a
>    couple of custom format functions that I wrote ... this will
>    probably turn out to be the culprit.

Yes, this sounds plausible.  Asynchronous buffer generation would be
nice ..

Re: Concerning marks and the back end.

[Kai Großjohann]

Lloyd Zusman <ljz@asfast.com> writes:

> I'm talking about the case where all messages are already marked as
> read, and I want to enter the group.

Well, which messages do you want to see in that case?
-- 
Ambibibentists unite!

Re: Concerning marks and the back end.

[Kai Großjohann]

Lloyd Zusman <ljz@asfast.com> writes:

> Nonetheless, I'm wondering if there's a way to mark articles in such a
> way that these marks are stored in the .marks file (instead of the back
> end), so that articles marked in this way will never be queried from the
> back end unless some special command is typed.

Read articles are never queried except in special
circumstances...  And you have hit them.

Maybe it would help if Gnus asked you about how many articles to
fetch.  Does it do that if gnus-large-newsgroup is small enough?
-- 
Ambibibentists unite!

Re: Concerning marks and the back end.

[Lloyd Zusman]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> I'm talking about the case where all messages are already marked as
>> read, and I want to enter the group.
>
> Well, which messages do you want to see in that case?

Like I said originally, I'd like to be able to put a special mark
(managed through the .marks file, not the back end) on my articles, such
that all of those already read articles that have this special mark are
NOT seen; and such that all of those already read articles WITHOUT that
special mark are still visible.


> -- 
> Ambibibentists unite!
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Lloyd Zusman]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> Nonetheless, I'm wondering if there's a way to mark articles in such a
>> way that these marks are stored in the .marks file (instead of the back
>> end), so that articles marked in this way will never be queried from the
>> back end unless some special command is typed.
>
> Read articles are never queried except in special
> circumstances...  And you have hit them.
>
> Maybe it would help if Gnus asked you about how many articles to
> fetch.  Does it do that if gnus-large-newsgroup is small enough?

Well, ideally I wouldn't get prompted, which is what happens in the
`gnus-large-newsgroup' logic.  But a prompt is better than nothing, I
guess ...


> -- 
> Ambibibentists unite!
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Kai Großjohann]

Lloyd Zusman <ljz@asfast.com> writes:

> Like I said originally, I'd like to be able to put a special mark
> (managed through the .marks file, not the back end) on my articles, such
> that all of those already read articles that have this special mark are
> NOT seen; and such that all of those already read articles WITHOUT that
> special mark are still visible.

You could tick the articles you want to see, and mark as read those
you don't want to see.

You could also mark as dormant the articles you want to see.  Then `Y
d' will show the dormant articles.

You can frob gnus-select-article-hook and/or gnus-mark-article-hook
to put marks other than `read' onto articles that you read.

-- 
Ambibibentists unite!

Re: Concerning marks and the back end.

[Lloyd Zusman]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> Like I said originally, I'd like to be able to put a special mark
>> (managed through the .marks file, not the back end) on my articles, such
>> that all of those already read articles that have this special mark are
>> NOT seen; and such that all of those already read articles WITHOUT that
>> special mark are still visible.
>
> You could tick the articles you want to see, and mark as read those
> you don't want to see.

Did you try this before suggesting it?

A tick mark doesn't help.  If you've been following this thread, you'll
know that I'm talking about the case where I enter a group with no
unread articles.  All the ticked and non-ticked articles show up when I
enter a group with no unread articles.


> You could also mark as dormant the articles you want to see.  Then `Y
> d' will show the dormant articles.

Thanks.  Soon I'll be off to investigate dormant-izing the articles.
I'll report back a little later.


> You can frob gnus-select-article-hook and/or gnus-mark-article-hook
> to put marks other than `read' onto articles that you read.

Would those marks get stored in the back end, or in the .marks file?
If they're stored in the back end, I don't know that they would
help me.


> -- 
> Ambibibentists unite!
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Lloyd Zusman]

Lloyd Zusman <ljz@asfast.com> writes:

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>
>> [ ... ]
>>
>> You could also mark as dormant the articles you want to see.  Then `Y
>> d' will show the dormant articles.
>
> Thanks.  Soon I'll be off to investigate dormant-izing the articles.
> I'll report back a little later.

... and now, I hereby report that this doesn't work, either.  When
entering a group with no unread articles, all dormant, ticked,
non-dormant, and non-ticked articles show up.



-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Lars Magne Ingebrigtsen]

Lloyd Zusman <ljz@asfast.com> writes:

>>> Like I said originally, I'd like to be able to put a special mark
>>> (managed through the .marks file, not the back end) on my articles, such
>>> that all of those already read articles that have this special mark are
>>> NOT seen; and such that all of those already read articles WITHOUT that
>>> special mark are still visible.
>>
>> You could tick the articles you want to see, and mark as read those
>> you don't want to see.
>
> Did you try this before suggesting it?
>
> A tick mark doesn't help.  If you've been following this thread, you'll
> know that I'm talking about the case where I enter a group with no
> unread articles.  All the ticked and non-ticked articles show up when I
> enter a group with no unread articles.

I don't quite follow you here.  You have:

1) Articles that are read that you want to see

2) Articles that are read that you don't want to see

3) New articles

Ticking the articles in 1) will achieve exactly this.  If there are
no unread articles in the group, only the ticked articles will be
shown when you enter the group.

(Unless you have customized a different behavior, of course.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Concerning marks and the back end.

[Lloyd Zusman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>>>> Like I said originally, I'd like to be able to put a special mark
>>>> (managed through the .marks file, not the back end) on my articles, such
>>>> that all of those already read articles that have this special mark are
>>>> NOT seen; and such that all of those already read articles WITHOUT that
>>>> special mark are still visible.
>>>
>>> You could tick the articles you want to see, and mark as read those
>>> you don't want to see.
>>
>> Did you try this before suggesting it?
>>
>> A tick mark doesn't help.  If you've been following this thread, you'll
>> know that I'm talking about the case where I enter a group with no
>> unread articles.  All the ticked and non-ticked articles show up when I
>> enter a group with no unread articles.
>
> I don't quite follow you here.  You have:
>
> 1) Articles that are read that you want to see
>
> 2) Articles that are read that you don't want to see
>
> 3) New articles

Not number 3.  No new articles.


> Ticking the articles in 1) will achieve exactly this.  If there are
> no unread articles in the group, only the ticked articles will be
> shown when you enter the group.
>
> (Unless you have customized a different behavior, of course.)

Well, maybe I have done this.  What variables control this behavior?

(Note that some of the code in my .gnus.el is probably more than 6 years
old, and I confess that I forgot what half of it does).

In my case, the behaviour is different depending on whether or not I
have new articles:

Case 1:  There are new articles when I enter a group.

  In this case, only the new and ticked articles show up.

Case 2:  There are no new articles when I enter a group.

  In this case, all ticked and unticked articles show up.


-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Lars Magne Ingebrigtsen]

Lloyd Zusman <ljz@asfast.com> writes:

> Well, maybe I have done this.  What variables control this behavior?

Do you have a `display' group parameter on the group, for instance?
There are probably more variables that control which articles are
selected...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Concerning marks and the back end.

[Lloyd Zusman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> Well, maybe I have done this.  What variables control this behavior?
>
> Do you have a `display' group parameter on the group, for instance?
> There are probably more variables that control which articles are
> selected...

Yes, I have "display", but it doesn't do what I want (see earlier part
of this thread, my discussion with Simon Josefsson).  I can set a
numeric value in "display", and here's what happens (suppose that the
numeric value is 100, that I have no ticked or dormant articles, and I
have several thousand already-read messages) ...

Case 1:  There are new articles when I enter the group (assume 10
         new articles for the moment).

  I see 10 new articles and 90 already read articles.

Case 2:  There are no new articles when I enter the group.

  I see 100 already read articles.

The second case is what I want.  But the first case isn't.  In Case 1,
I'd like to only see the new articles.
  
Can anyone think of any other variables that control this behavior?


> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    larsi@gnus.org * Lars Magne Ingebrigtsen
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Paul Jarc]

Lloyd Zusman <ljz@asfast.com> wrote:
> Would those marks get stored in the back end, or in the .marks file?

There's no difference.  The .marks file is precisely the backend's
mark storage.


paul

Re: Concerning marks and the back end.

[Lloyd Zusman]

prj@po.cwru.edu (Paul Jarc) writes:

> Lloyd Zusman <ljz@asfast.com> wrote:
>> Would those marks get stored in the back end, or in the .marks file?
>
> There's no difference.  The .marks file is precisely the backend's
> mark storage.

Oh ... thanks.  Until this moment, I had always been confused about
that.

Well then, all my comments about the marks having to be in .marks are
now null and void. :)

... but I still would like to achieve my original goal.


> paul


-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Kai Großjohann]

Lloyd Zusman <ljz@asfast.com> writes:

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>
>> Lloyd Zusman <ljz@asfast.com> writes:
>>
>>> Like I said originally, I'd like to be able to put a special mark
>>> (managed through the .marks file, not the back end) on my articles, such
>>> that all of those already read articles that have this special mark are
>>> NOT seen; and such that all of those already read articles WITHOUT that
>>> special mark are still visible.
>>
>> You could tick the articles you want to see, and mark as read those
>> you don't want to see.
>
> Did you try this before suggesting it?
>
> A tick mark doesn't help.  If you've been following this thread, you'll
> know that I'm talking about the case where I enter a group with no
> unread articles.  All the ticked and non-ticked articles show up when I
> enter a group with no unread articles.

Note that I suggested to tick the articles you want to see.  So the
tick mark would be the opposite of what you're looking for.

>> You could also mark as dormant the articles you want to see.  Then `Y
>> d' will show the dormant articles.
>
> Thanks.  Soon I'll be off to investigate dormant-izing the articles.
> I'll report back a little later.

Dormant-izing the ones you want to see, that is.

>> You can frob gnus-select-article-hook and/or gnus-mark-article-hook
>> to put marks other than `read' onto articles that you read.
>
> Would those marks get stored in the back end, or in the .marks file?

I think you mean XOR here?

Note that the .marks file is part of the backend.  So the marks would
be stored in .marks and therefore in the backend, so the answer would
be "both".

(Not all backends use .marks files.)

> If they're stored in the back end, I don't know that they would
> help me.

???  Why does it matter where they are stored?

-- 
Ambibibentists unite!

Re: Concerning marks and the back end.

[Kai Großjohann]

Lloyd Zusman <ljz@asfast.com> writes:

> ... and now, I hereby report that this doesn't work, either.  When
> entering a group with no unread articles, all dormant, ticked,
> non-dormant, and non-ticked articles show up.

Try `1 RET'.  You see one read message.  Now do `Y d'.  Now you see
one read message plus a number of dormant ones.

Doing `Y d' can be done from Lisp.
-- 
Ambibibentists unite!

Re: Concerning marks and the back end.

[Kai Großjohann]

Lloyd Zusman <ljz@asfast.com> writes:

> Case 2:  There are no new articles when I enter a group.
>
>   In this case, all ticked and unticked articles show up.

This does not happen for me.  Because of gnus-fetch-old-headers,
_some_ unticked articles could show up, but usually not all of them.
-- 
Ambibibentists unite!

Re: Concerning marks and the back end.

[Kai Großjohann]

Lloyd Zusman <ljz@asfast.com> writes:

> The second case is what I want.  But the first case isn't.  In Case 1,
> I'd like to only see the new articles.
>   
> Can anyone think of any other variables that control this behavior?

You should remove the display parameter for my "tick" suggestion to work.

-- 
Ambibibentists unite!

Re: Concerning marks and the back end.

[Lloyd Zusman]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> ... and now, I hereby report that this doesn't work, either.  When
>> entering a group with no unread articles, all dormant, ticked,
>> non-dormant, and non-ticked articles show up.
>
> Try `1 RET'.  You see one read message.  Now do `Y d'.  Now you see
> one read message plus a number of dormant ones.

This is the suggestion that Lars made yesterday.

But I don't want to type one set of keystrokes to enter a group with
unread articles, and a different set of keystrokes to enter a group with
no unread articles.

If I type the same keystrokes for both cases, here's what happens ...

Case 1:  Group has 5 unread articles.  I type 10 RET.

  I see 5 unread articles and 5 already-read articles.

Case 2:  Group has no unread articles.  I type 10 RET.

  I see 10 unread articles.

I want Case 2.  I don't want Case 1.  In the first case, I only want to
see the 5 unread articles.


> Doing `Y d' can be done from Lisp.

Yep.


-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Lloyd Zusman]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> Case 2:  There are no new articles when I enter a group.
>>
>>   In this case, all ticked and unticked articles show up.
>
> This does not happen for me.  Because of gnus-fetch-old-headers,
> _some_ unticked articles could show up, but usually not all of them.

For me, gnus-fetch-old-headers has been nil.  I set it to the integer
100, but that didn't change anything that happens when I enter a group
... I still have the "Case 1" and "Case 2" behavior that I described
in my earlier messages, irrespective of the gnus-fetch-old-headers
setting.


-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Kai Großjohann]

Lloyd Zusman <ljz@asfast.com> writes:

> But I don't want to type one set of keystrokes to enter a group with
> unread articles, and a different set of keystrokes to enter a group with
> no unread articles.

Now that you have all the pieces, you can just write a function to
call instead of gnus-topic-select-group that looks if the group is
empty and then does one thing if yes and another thing if no.  No?
-- 
Ambibibentists unite!

Re: Concerning marks and the back end.

[Lloyd Zusman]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> But I don't want to type one set of keystrokes to enter a group with
>> unread articles, and a different set of keystrokes to enter a group with
>> no unread articles.
>
> Now that you have all the pieces, you can just write a function to
> call instead of gnus-topic-select-group that looks if the group is
> empty and then does one thing if yes and another thing if no.  No?

That's exactly what I said I was going to do (read the earlier portion
of this thread).

> -- 
> Ambibibentists unite!
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Concerning marks and the back end.

[Lloyd Zusman]

Lloyd Zusman <ljz@asfast.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>>
>> [ ... ]
>>
>> You can set the display parameter to an array of predicates, e.g.:
>>
>> [or unread (and no-unreads my-article-old-p)]
>>
>>      (defun my-article-old-p ()
>>        "Say whether an article is old."
>>        (< (time-to-days (date-to-time (mail-header-date gnus-headers)))
>>           (- (time-to-days (current-time)) gnus-agent-expire-days)))
>>
>> implementing the no-unreads function is left as an exercise. :-)
>>
>> But this might not work either, as I think display predicates was
>> implemented as limiting, so you will fetch all article headers anyway.
>
> Hmmm ... well, I could always cheat and use `defadvice' to wrap the
> function which enters the group; [ ... ]
>
> Sounds easy, anyway. :)

... and indeed it turned out to be straightforward.  The defadvice'd
function is below.  Yes, I know it's an ugly hack, and I'm still hoping
for a better way to do this.  But at least this works in the mean time.


>> But I still wonder why entering the group is slow for you.  I have
>> 15,000 messages in my ding mailing list mailbox, which is stored on a
>> IMAP server, and entering it and viewing the unread and ticked
>> articles takes no time.  When I get several hundred thousand articles
>> it becomes a little slow though (long coffe break).
>
> Well, I still will do the profiling that you suggested earlier.

... and I'll do this profiling soon.

Here's the function.  I could have used fewer variable definition steps
in the 'let' clause, but I broke it into more pieces for ease of
debugging during development.


 (defvar ljz-group-default-display-count 256
   "*Default count of articles to display when entering a group, if there
 are no unread, dormant, or ticked articles, and if there is no prefix
 argument.")

 (defadvice gnus-group-read-group (around gnus-group-read-group freeze)
   (interactive "P")
   (let* ((the-group (or group (gnus-group-group-name)))
	  (the-unread-count (gnus-group-unread the-group))
	  (the-marks (gnus-info-marks (gnus-get-info the-group)))
	  (the-dormant (cdr (assq 'dormant the-marks)))
	  (the-ticked (cdr (assq 'tick the-marks)))
	  (the-count
	   (+ the-unread-count (length the-dormant) (length the-ticked)))
	  (the-parameter (if all all (if (> the-count 0)
					 nil
				       ljz-group-default-display-count))))
     (ad-set-arg 0 the-parameter)
     ad-do-it))


-- 
 Lloyd Zusman
 ljz@asfast.com

A solution that's better than my ugly hack?

[Lloyd Zusman]

A week or two ago I posted this ugly hack here (quoted below) to provide
a feature that I've wanted: if I enter a group with no unread articles,
I only see the last N already-read articles (where N is configurable),
instead of the entire set of already-read's; but if I enter a group with
at least one unread article, I only see the unreads (and in both cases,
I'd also see the ticked and dormant articles).

Has anyone been able to figure out a cleaner way of getting this same
functionality, instead of this horrible, defadvice-based hack?

Thanks in advance.


>  (defvar ljz-group-default-display-count 256
>    "*Default count of articles to display when entering a group, if there
>  are no unread, dormant, or ticked articles, and if there is no prefix
>  argument.")
>
>  ;; Yes, I know that all of the variables defined in the `let*' clause
>  ;; are not really necessary, but I put them there so I could more easily
>  ;; put meaningful debug statements into the code during development.
>  (defadvice gnus-group-read-group (around gnus-group-read-group freeze)
>    (interactive "P")
>    (let* ((the-group (or group (gnus-group-group-name)))
>           (the-unread-count (gnus-group-unread the-group))
>           (the-marks (gnus-info-marks (gnus-get-info the-group)))
>           (the-dormant (cdr (assq 'dormant the-marks)))
>           (the-ticked (cdr (assq 'tick the-marks)))
>           (the-count
> 	      (+ the-unread-count (length the-dormant) (length the-ticked)))
>           (the-parameter (if all all (if (> the-count 0)
> 					 nil
> 				       ljz-group-default-display-count))))
>      (ad-set-arg 0 the-parameter)
>      ad-do-it))
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: A solution that's better than my ugly hack?

[Lars Magne Ingebrigtsen]

Lloyd Zusman <ljz@asfast.com> writes:

> Has anyone been able to figure out a cleaner way of getting this same
> functionality, instead of this horrible, defadvice-based hack?

Well, there is the `gnus-alter-articles-to-read-function' variable, 
which I think you can use to, er, alter the list of articles to
read.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: A solution that's better than my ugly hack?

[Lloyd Zusman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> Has anyone been able to figure out a cleaner way of getting this same
>> functionality, instead of this horrible, defadvice-based hack?
>
> Well, there is the `gnus-alter-articles-to-read-function' variable, 
> which I think you can use to, er, alter the list of articles to
> read.  :-)

Thank you.  This is a new one for me.

But unless I'm missing something, it looks like this function only gets
applied to the list of unread articles.  Is that correct?  I think that
I would need a similar function that would get applied to the list of
already-read articles.

My desired functionality is to shorten the list of already-read articles
that would get displayed, but to list the unreads as they normally get
listed.


-- 
 Lloyd Zusman
 ljz@asfast.com

Re: A solution that's better than my ugly hack?

[Lars Magne Ingebrigtsen]

Lloyd Zusman <ljz@asfast.com> writes:

> But unless I'm missing something, it looks like this function only gets
> applied to the list of unread articles.  Is that correct?  I think that
> I would need a similar function that would get applied to the list of
> already-read articles.

Yes, and that doesn't quite line up with the documentation of the
variable.

I'll apply the following patch:

*** gnus-sum.el.~6.283.~	Mon Jan 13 19:35:35 2003
--- gnus-sum.el	Mon Jan 13 20:17:27 2003
***************
*** 5212,5221 ****
        (setq gnus-newsgroup-unselected
  	    (gnus-sorted-difference gnus-newsgroup-unreads articles))
        (when gnus-alter-articles-to-read-function
! 	(setq gnus-newsgroup-unreads
  	      (sort
  	       (funcall gnus-alter-articles-to-read-function
! 			gnus-newsgroup-name gnus-newsgroup-unreads)
  	       '<)))
        articles)))
  
--- 5212,5221 ----
        (setq gnus-newsgroup-unselected
  	    (gnus-sorted-difference gnus-newsgroup-unreads articles))
        (when gnus-alter-articles-to-read-function
! 	(setq articles
  	      (sort
  	       (funcall gnus-alter-articles-to-read-function
! 			gnus-newsgroup-name articles)
  	       '<)))
        articles)))
  


-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: A solution that's better than my ugly hack?

[Lloyd Zusman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> But unless I'm missing something, it looks like this function only gets
>> applied to the list of unread articles.  Is that correct?  I think that
>> I would need a similar function that would get applied to the list of
>> already-read articles.
>
> Yes, and that doesn't quite line up with the documentation of the
> variable.
>
> I'll apply the following patch:

Thank you!

This looks like it would do the trick.

And this brings up one more question: for the list of articles that gets
passed to `gnus-alter-articles-to-read-function', how do I tell which
of these articles are unread, and which are already read?

Is there perhaps a function I can call with an article number as an
argument which will tell me if it's read/unread/whatever?  Or are there
appropriate hashtables that I can access with the article number as a
key?  Or ... ???


> *** gnus-sum.el.~6.283.~	Mon Jan 13 19:35:35 2003
> --- gnus-sum.el	Mon Jan 13 20:17:27 2003
> ***************
> *** 5212,5221 ****
>         (setq gnus-newsgroup-unselected
>   	    (gnus-sorted-difference gnus-newsgroup-unreads articles))
>         (when gnus-alter-articles-to-read-function
> ! 	(setq gnus-newsgroup-unreads
>   	      (sort
>   	       (funcall gnus-alter-articles-to-read-function
> ! 			gnus-newsgroup-name gnus-newsgroup-unreads)
>   	       '<)))
>         articles)))
>   
> --- 5212,5221 ----
>         (setq gnus-newsgroup-unselected
>   	    (gnus-sorted-difference gnus-newsgroup-unreads articles))
>         (when gnus-alter-articles-to-read-function
> ! 	(setq articles
>   	      (sort
>   	       (funcall gnus-alter-articles-to-read-function
> ! 			gnus-newsgroup-name articles)
>   	       '<)))
>         articles)))
>   
>
>
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    larsi@gnus.org * Lars Magne Ingebrigtsen
>

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: A solution that's better than my ugly hack?

[Lars Magne Ingebrigtsen]

Lloyd Zusman <ljz@asfast.com> writes:

> And this brings up one more question: for the list of articles that gets
> passed to `gnus-alter-articles-to-read-function', how do I tell which
> of these articles are unread, and which are already read?

(memq article gnus-newsgroup-reads)

and stuff like that should do the trick.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: A solution that's better than my ugly hack?

[Lloyd Zusman]

Lloyd Zusman <ljz@asfast.com> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>
>> [ ... ]
>>
>> Yes, and that doesn't quite line up with the documentation of the
>> variable.
>>
>> I'll apply the following patch:
>
> Thank you!
>
> This looks like it would do the trick.
>
> And this brings up one more question: for the list of articles that gets
> passed to `gnus-alter-articles-to-read-function', how do I tell which
> of these articles are unread, and which are already read?
>
> Is there perhaps a function I can call with an article number as an
> argument which will tell me if it's read/unread/whatever?  Or are there
> appropriate hashtables that I can access with the article number as a
> key?  Or ... ???

Replying to my own post: I think I found the answer.  I could use the
macros `gnus-data-read-p', `gnus-data-unread-p', and `gnus-data-mark',
correct?


> [ ... ]

-- 
 Lloyd Zusman
 ljz@asfast.com

Re: A solution that's better than my ugly hack?

[Lloyd Zusman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> And this brings up one more question: for the list of articles that gets
>> passed to `gnus-alter-articles-to-read-function', how do I tell which
>> of these articles are unread, and which are already read?
>
> (memq article gnus-newsgroup-reads)
>
> and stuff like that should do the trick.

Ah ... got it.  Thanks.



-- 
 Lloyd Zusman
 ljz@asfast.com

Re: A solution that's better than my ugly hack?

[Lloyd Zusman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lloyd Zusman <ljz@asfast.com> writes:
>
>> And this brings up one more question: for the list of articles that gets
>> passed to `gnus-alter-articles-to-read-function', how do I tell which
>> of these articles are unread, and which are already read?
>
> (memq article gnus-newsgroup-reads)
>
> and stuff like that should do the trick.

And it indeed did do the trick.  Here's my solution:

 (defvar ljz-group-default-display-count 256
   "*Default count of articles to display when entering a group, if there
 are no unread, dormant, or ticked articles, and if there is no prefix
 argument.")

 (defun ljz-alter-articles-to-read-function (group article-list)
   (let* ((count 0)
          (prefix (if (listp current-prefix-arg)
                      (car current-prefix-arg)
                    current-prefix-arg))
          (test-count (if prefix prefix ljz-group-default-display-count))
          newlist article)
     (if (or (null test-count) (<= test-count 0))
         article-list
       (while article-list
         (setq article (car article-list))
         (setq article-list (cdr article-list))
         (when (or (memq article gnus-newsgroup-unreads)
                   (memq article gnus-newsgroup-marked)
                   (memq article gnus-newsgroup-dormant)
                   (<= (setq count (1+ count )) test-count))
           (setq newlist (append newlist (list article)))))
       newlist)))

I then customized `gnus-alter-articles-to-read-function' to contain
'ljz-alter-articles-to-read-function, and it works great!

Thanks again.

-- 
 Lloyd Zusman
 ljz@asfast.com