`A T' and Gmanes humongus groups

`A T' and Gmanes humongus groups

[Harry Putnam]

Considering Lars nifty policy of never expunging gmane groups, I
woundered if it might be in order to add a modified `A T' command to
gnus.  Or maybe its' already doable some other way.

I'm thinking  it might be possible to somehow tie a date to `A T' so
one could say `C-u 2 A T' and parse only 2wks back instead of all
$LARGNUMBER messages.

Re: `A T' and Gmanes humongus groups

[Lars Magne Ingebrigtsen]

Harry Putnam <hgp@sbcglobal.net> writes:

> I'm thinking  it might be possible to somehow tie a date to `A T' so
> one could say `C-u 2 A T' and parse only 2wks back instead of all
> $LARGNUMBER messages.

You can't know in advance how many headers you have to fetch to get
two week's worth of headers.

`C-u 1000 A T' will fetch the previous 1000 headers.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: `A T' and Gmanes humongus groups

[Harry Putnam]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Harry Putnam <hgp@sbcglobal.net> writes:
>
>> I'm thinking  it might be possible to somehow tie a date to `A T' so
>> one could say `C-u 2 A T' and parse only 2wks back instead of all
>> $LARGNUMBER messages.
>
> You can't know in advance how many headers you have to fetch to get
> two week's worth of headers.
>
> `C-u 1000 A T' will fetch the previous 1000 headers.

Yeah, thats what I was looking for... thanks.

Re: `A T' and Gmanes humongus groups

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Harry Putnam <hgp@sbcglobal.net> writes:
>
>> I'm thinking  it might be possible to somehow tie a date to `A T' so
>> one could say `C-u 2 A T' and parse only 2wks back instead of all
>> $LARGNUMBER messages.
>
> You can't know in advance how many headers you have to fetch to get
> two week's worth of headers.
>
> `C-u 1000 A T' will fetch the previous 1000 headers.

I wonder if a chunked mode might be useful?  Fetch $chunksize
headers, look if the root of the thread is in there.  If not, fetch
another $chunksize headers.  Lather, rinse, repeat.

-- 
Ambibibentists unite!

Re: `A T' and Gmanes humongus groups

[Lars Magne Ingebrigtsen]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> I wonder if a chunked mode might be useful?  Fetch $chunksize
> headers, look if the root of the thread is in there.  If not, fetch
> another $chunksize headers.  Lather, rinse, repeat.

That would be a possibility, but there would have to be a limit to
how far back one should go -- the thread root might not be on the
server (or in the group) at all.

Chunking would make most sense for groups from servers like nntp and
nnimap -- chunking for nnml, for instance, would probably not help
much.  So would this best be placed as a new interface function?
`nnchoke-retrieve-headers-until-message-id'?  If that function isn't
implemented in the back end, it could fall back on using
`nnchoke-retrieve-headers'.

Sounds ok?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: `A T' and Gmanes humongus groups

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Chunking would make most sense for groups from servers like nntp and
> nnimap -- chunking for nnml, for instance, would probably not help
> much.  So would this best be placed as a new interface function?
> `nnchoke-retrieve-headers-until-message-id'?  If that function isn't
> implemented in the back end, it could fall back on using
> `nnchoke-retrieve-headers'.

Good idea.

-- 
Ambibibentists unite!

Re: `A T' and Gmanes humongus groups

[Lars Magne Ingebrigtsen]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

>> Chunking would make most sense for groups from servers like nntp and
>> nnimap -- chunking for nnml, for instance, would probably not help
>> much.  So would this best be placed as a new interface function?
>> `nnchoke-retrieve-headers-until-message-id'?  If that function isn't
>> implemented in the back end, it could fall back on using
>> `nnchoke-retrieve-headers'.
>
> Good idea.

I was afraid you'd say that.  :-)

I've now experimented a bit with the function `A T' command, and I
found that for me, the main problem was this code:

    ;; We want to fetch LIMIT *old* headers, but we also have to
    ;; re-fetch all the headers in the current buffer, because many of
    ;; them may be undisplayed.  So we adjust LIMIT.
    (when (numberp limit)
      (incf limit (- gnus-newsgroup-end gnus-newsgroup-begin))))

In some groups I have dormant old articles, and this means that this
will fetch an enormous number of messages.  I've just removed this,
and had it fetch the previous/next 200 messages in the group.  This
seems to do the trick most of the time, actually. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: `A T' and Gmanes humongus groups

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> In some groups I have dormant old articles, and this means that this
> will fetch an enormous number of messages.  I've just removed this,
> and had it fetch the previous/next 200 messages in the group.  This
> seems to do the trick most of the time, actually. 

Wouldn't it be much cleverer to let the server do the dirty searching
job and make a XPAT search on references header?

(This could give the user support for xpat searches I've been whining
for so long, too :-))

-- 
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.

Re: `A T' and Gmanes humongus groups

[Lars Magne Ingebrigtsen]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> Wouldn't it be much cleverer to let the server do the dirty searching
> job and make a XPAT search on references header?

Do news servers support XPAT these days?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: `A T' and Gmanes humongus groups

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:
>
>> Wouldn't it be much cleverer to let the server do the dirty searching
>> job and make a XPAT search on references header?
>
> Do news servers support XPAT these days?

Those I use do, (INN 2.3 and Hamster).

-- 
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.

Re: `A T' and Gmanes humongus groups

[Lars Magne Ingebrigtsen]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> Those I use do, (INN 2.3 and Hamster).

I'm too lazy to Google for the syntax today.  What's the syntax for
using XPAT to query a news server for Message-IDs that match a string?

(Anyway, this won't really do the trick reliably.  News clients are
free to drop ids from references, and they do.  But that's a
quality/speed issue.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: `A T' and Gmanes humongus groups

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm too lazy to Google for the syntax today.  What's the syntax for
> using XPAT to query a news server for Message-IDs that match a string?

It looks like this:

xpat References 5921 6195 *userfqdn*
221 Header follows
5921 <uof6v23v9.fsf@ichimusai.org> <87vg13smd0.fsf@kuntu.zangpo.org>
 <87k7hjven1.fsf@at_ipipan.waw.pl>
 <4r8nz8te.fsf@hschmi22.userfqdn.rz-online.de>
 <u4r8n5mbh.fsf@ichimusai.org>
.

-- 
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.

Re: `A T' and Gmanes humongus groups

[Lars Magne Ingebrigtsen]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> xpat References 5921 6195 *userfqdn*

That doesn't give me any results on Quimby, apparently.  (Which runs
a pretty default Potato Debian inn.)  So perhaps it isn't switched on
by default?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: `A T' and Gmanes humongus groups

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:
>
>> xpat References 5921 6195 *userfqdn*
>
> That doesn't give me any results on Quimby, apparently.  (Which runs
> a pretty default Potato Debian inn.)  So perhaps it isn't switched on
> by default?

It is, I just had the wrong memory and a server which gratefully
accepted the wrong command.

You can find the detailed description for XPat at
http://www.mibsoftware.com/userkt/nntpext/0026.htm

in the example above, it should be 
xpat References 5921-6195 *userfqdn*

It works on Quimby, too:

200 quimby.gnus.org InterNetNews NNRP server INN 2.2.2 13-Dec-1999 ready (posting ok).
group gnus.ding
211 45586 3 45777 gnus.ding
xpat references 45000-45777 *userfqdn*
221 references matches follow.
45040 <848yy2azri.fsf@lucy.cs.uni-dortmund.de> <r8bszyy9.fsf@hschmi22.userfqdn.rz-online.de>
45144 <848yy2azri.fsf@lucy.cs.uni-dortmund.de> <r8bszyy9.fsf@hschmi22.userfqdn.rz-online.de>
45158 <848yy2azri.fsf@lucy.cs.uni-dortmund.de> <r8bszyy9.fsf@hschmi22.userfqdn.rz-online.de>
45275 <848yy2azri.fsf@lucy.cs.uni-dortmund.de> <r8bszyy9.fsf@hschmi22.userfqdn.rz-online.de>
...

-- 
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.

Re: `A T' and Gmanes humongus groups

[Lars Magne Ingebrigtsen]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> in the example above, it should be 
> xpat References 5921-6195 *userfqdn*
>
> It works on Quimby, too:

So it does.  So the return value is basically a list of article
numbers (with some more text at the end of the lines).  That can be
used, but I think that the current `A T' implementation will most
likely give better results in most cases (as noted in the previous
message). 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: `A T' and Gmanes humongus groups

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:
>
>> in the example above, it should be 
>> xpat References 5921-6195 *userfqdn*
>>
>> It works on Quimby, too:
>
> So it does.  So the return value is basically a list of article
> numbers (with some more text at the end of the lines).  That can be
> used, but I think that the current `A T' implementation will most
> likely give better results in most cases (as noted in the previous
> message). 

I don't think so, as well behaving news readers don't drop the MID of the
first article. (Isn't this a "should" in some standard?)

-- 
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.

Re: `A T' and Gmanes humongus groups

[Lars Magne Ingebrigtsen]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> I don't think so, as well behaving news readers don't drop the MID of the
> first article. (Isn't this a "should" in some standard?)

There used to be, but if I recall correctly, this is no longer a
requirement.  Clients are free to trim any and all Message-IDs from
the References header (except the final one, of course).

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen