Unable to create Dejanews nnweb groups

Unable to create Dejanews nnweb groups

[Hannu Koivisto]

Greetings,

[Happens with both 0.96 and CVS version checked out at about
1999-11-09T17:21+0200.]

To save myself from the horrors of Dejanews' WWW site I decided to
try Gnus' search engine interface in an attempt to locate articles
mentioning "clorb" that have appeared in the comp.lang.lisp
newsgroup.

`G w dejanews<RET> clorb<RET>' results in a message "Contacting
x8.dejanews..." and after that a question "Honor redirection with
non-GET method (possible security risks)? (y or n) " to which I
answered `y' as I did to a following identical question.  After
that, Gnus says it contacts `www.deja.com:80', thinks for awhile,
and then says "Couldn't request group: No matching articles".

If I specify a newsgroup in search string (i.e. "~g comp.lang.lisp
clorb" as instructed in the Gnus manual), I get exactly the same
result.

If I use Dejanews <URL:http://www.deja.com/> directly, I get two
articles if I simply enter "clorb" to that quick search thing on
their front page and five articles (this time one of them is
actually the one I was looking for!) if I go to that "power search"
thing and enter "clorb" to the keywords box and "comp.lang.lisp" to
the forum box.

Am I doing something wrong or have they changed their output
(again) and Gnus hasn't yet been modified accordingly?

-- 
Hannu

Re: Unable to create Dejanews nnweb groups

[Steinar Bang]

To anyone wishing to update the Deja News HTML parsing: note that you
can get at the old HTML (that nnweb presumably parses) by adding
"/=dnc" beween the machine name of the query URL, and the first "/" of
the path name of the URL.

Re: Unable to create Dejanews nnweb groups

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@metis.no> writes:

> To anyone wishing to update the Deja News HTML parsing: note that you
> can get at the old HTML (that nnweb presumably parses) by adding
> "/=dnc" beween the machine name of the query URL, and the first "/" of
> the path name of the URL.

Well, it may look similar, but the HTML is different.  In CVS now is
the correct URLs and stuff, but I have't updated to do the right
thing.  It shouldn't be too much work, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Unable to create Dejanews nnweb groups

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@metis.no> writes:
>> To anyone wishing to update the Deja News HTML parsing: note that you
>> can get at the old HTML (that nnweb presumably parses) by adding
>> "/=dnc" beween the machine name of the query URL, and the first "/" of
>> the path name of the URL.

> Well, it may look similar, but the HTML is different.  In CVS now is
> the correct URLs and stuff, but I have't updated to do the right
> thing.  It shouldn't be too much work, though.

Something I forgot: both Deja Classic and the new Deja is able to
return the news articles as plain text.

Here's one of mine:
        http://www.deja.com/=dnc/getdoc.xp?AN=487815218&fmt=text
(Deja classic)
        http://www.deja.com/=rd/getdoc.xp?AN=487815218&fmt=text
(ordinary Deja)

Re: Unable to create Dejanews nnweb groups

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@metis.no> writes:

> Something I forgot: both Deja Classic and the new Deja is able to
> return the news articles as plain text.
> 
> Here's one of mine:
>         http://www.deja.com/=dnc/getdoc.xp?AN=487815218&fmt=text

Supercool!  With total and complete headers and all!

nnweb now does dejanews searches ok again.

Wasn't there some URL where you could request articles by Message-ID
from Deja?  I want to use that as my `gnus-refer-article-method'.
What's the URL?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Unable to create Dejanews nnweb groups

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Wasn't there some URL where you could request articles by Message-ID
> from Deja?  I want to use that as my `gnus-refer-article-method'.
> What's the URL?

I found it; it was in the TODO section in the manual.  My, there's a
lot of stuff in there.

So I've now implemented it.

(setq gnus-refer-article-method '(nnweb "refer" (nnweb-type dejanews)))

It in CVS.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Unable to create Dejanews nnweb groups

[Per Abrahamsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So I've now implemented it.
> 
> (setq gnus-refer-article-method '(nnweb "refer" (nnweb-type dejanews)))
> 
> It in CVS.

I made DejaNews a customization option.

I think the docstring ought to be improved, I couldn't figure out how
it works.  For example, it isn't used in nnml buffer as far as I can
see. 

It should probably be a list, so you could tell it first to try the
default method, then some other nntp server, and then dejanews.  It
feel weird contacting dejanews just to get the parent in a local nntp
group.  The default behaviour could be first to try the default
method, and then DejaNews.

1999-11-12  Per Abrahamsen  <abraham@dina.kvl.dk>

	* gnus.el (gnus-select-method): Added default value and tag.
	(gnus-refer-article-method): Added `DejaNews' customization option.

Index: gnus.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus.el,v
retrieving revision 5.109
diff -c -r5.109 gnus.el
*** gnus.el	1999/11/10 19:05:56	5.109
--- gnus.el	1999/11/12 17:30:22
***************
*** 999,1004 ****
--- 999,1005 ----
  in the documentation of `gnus-select-method'."
    :group 'gnus-server
    :type '(choice (const :tag "default" nil)
+ 		 (const :tag "DejaNews" (nnweb "refer" (nnweb-type dejanews)))
  		 gnus-select-method))
  
  (defcustom gnus-group-faq-directory
***************
*** 1261,1266 ****
--- 1262,1269 ----
  
  (define-widget 'gnus-select-method 'list
    "Widget for entering a select method."
+   :value '(nntp "")
+   :tag "Select Method"
    :args `((choice :tag "Method"
  		  ,@(mapcar (lambda (entry)
  			      (list 'const :format "%v\n"

Re: Unable to create Dejanews nnweb groups

[Lars Magne Ingebrigtsen]

Per Abrahamsen <abraham@dina.kvl.dk> writes:

> It should probably be a list, so you could tell it first to try the
> default method, then some other nntp server, and then dejanews.  It
> feel weird contacting dejanews just to get the parent in a local nntp
> group.

I've now added this.

> The default behaviour could be first to try the default method, and
> then DejaNews.

That would mean that the default would be to go out on the net when
`^'-ing (and the current method fails), and I don't know how happy
people will be about that...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Unable to create Dejanews nnweb groups

[Rene H. Larsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Per Abrahamsen <abraham@dina.kvl.dk> writes:

[...]

> > The default behaviour could be first to try the default method, and
> > then DejaNews.
> 
> That would mean that the default would be to go out on the net when
> `^'-ing (and the current method fails), and I don't know how happy
> people will be about that...

What if Gnus were to first try the current method, and if that fails,
ask (y-or-n-p) if it should try DejaNews.  This could be customizable
by setting gnus-article-refer-try-dejanews to 'always, 'never, or
'ask, e.g.
-- 
Echelon bait:
cryptographic Paula Corbin Jones World Trade Center clones bomb Rule
Psix Marxist Ruby Ridge $400 million in gold bullion Noriega Roswell
PLO North Korea Delta Force Kibo

Re: Unable to create Dejanews nnweb groups

[Lars Magne Ingebrigtsen]

"Rene H. Larsen" <rhl@traceroute.dk> writes:

> What if Gnus were to first try the current method, and if that fails,
> ask (y-or-n-p) if it should try DejaNews.  This could be customizable
> by setting gnus-article-refer-try-dejanews to 'always, 'never, or
> 'ask, e.g.

Hm.  I'm not much of a fan of queries, really.  I think people should
just customize this to do what they want.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Unable to create Dejanews nnweb groups

[Rene H. Larsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> "Rene H. Larsen" <rhl@traceroute.dk> writes:
> 
> > What if Gnus were to first try the current method, and if that fails,
> > ask (y-or-n-p) if it should try DejaNews.  This could be customizable
> > by setting gnus-article-refer-try-dejanews to 'always, 'never, or
> > 'ask, e.g.
> 
> Hm.  I'm not much of a fan of queries, really.  I think people should
> just customize this to do what they want.

I can certainly sympathise with that in the general case.  The reason
I was looking for a way to make Gnus query whether to try Deja is that
I have an intermittent Internet connection, and there is no easy way
to tell if the Internet connection is up.  I don't want Gnus to try
and contact Deja when the connection is down, but I would like to have
the option to do so when the connection is up.

I did try mocking around with gnus-refer-article-method, and using the
example from the Info documentation, namely

     (setq gnus-refer-article-method
           '(current
             (nnweb "refer" (nnweb-type dejanews))))

does not seem to work correctly.  Pressing ^ on an article whose
parent is still present on the server works correctly.  However, if I
press ^ on an article whose parent has expired, XEmacs complains:

  Cannot open load file: current

The backtrace is:

  signal(file-error ("Cannot open load file" "current"))
  load("current" nil t nil)
  require(current)
  gnus-server-opened((current (nnweb "refer" (nnweb-type dejanews))))
  gnus-check-group-server()
  gnus-request-article-this-buffer(-4 "dk.edb.system.unix.hardware")
  gnus-article-prepare(-4 nil)
  gnus-summary-display-article(-4 nil)
  gnus-summary-goto-article(-4 nil t)
  gnus-summary-refer-article("<syvK3.1969$et6.51290@news010.image.dk>")
  gnus-summary-refer-parent-article(1)
  call-interactively(gnus-summary-refer-parent-article)

This is with a CVS pgnus as of ten minutes ago.
-- 
Echelon bait:
supercomputer Nazi AK-47 Bosnia Croatian Uzi SCUD missile Legion of
Doom White Water Marxist Ron Brown smuggle Craig Livingstone Project
Monarch quiche

Re: Unable to create Dejanews nnweb groups

[Lars Magne Ingebrigtsen]

"Rene H. Larsen" <rhl@traceroute.dk> writes:

> I can certainly sympathise with that in the general case.  The reason
> I was looking for a way to make Gnus query whether to try Deja is that
> I have an intermittent Internet connection, and there is no easy way
> to tell if the Internet connection is up.  I don't want Gnus to try
> and contact Deja when the connection is down, but I would like to have
> the option to do so when the connection is up.

In that case, I'd suggest fiddling with `gnus-refer-article-method' in
`gnus-agent-unplugged-hook'.

> I did try mocking around with gnus-refer-article-method, and using the
> example from the Info documentation, namely
> 
>      (setq gnus-refer-article-method
>            '(current
>              (nnweb "refer" (nnweb-type dejanews))))
> 
> does not seem to work correctly.

Yup.  Fix in Gnus v5.8.3.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Unable to create Dejanews nnweb groups

[Hannu Koivisto]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

| Supercool!  With total and complete headers and all!
| 
| nnweb now does dejanews searches ok again.

(loop Supercool!)  First I thought that the result is still not
quite correct, but now that I read the manual, the behaviour I'm
seeing seems to be a feature of nnweb.  That is, if I use my
original example again and use "~g comp.lang.lisp clorb" as the
search string, I get a summary buffer like this:

"""
R  [   0: Robert Monfera      ] Re: LispWorks status<clip>
R  [   0: David Hanley        ] Re: java frontend for<clip>
R  [   0: David Hanley        ] Re: Announcing CLORB a<clip>
R      <   0: Raymond Wiker       >
R      <   0: Lennart Staflin     >
"""

whereas Dejanews tells me (irrelevant stuff stripped; output is
from w3m):

"""
Date *   Subject *      Author 
10/12/   Re: Announcing David  
99       CLORB a Commo  Hanley 
10/10/   Announcing     Lennart
99       CLORB a Common Staflin
         Li                    
10/29/   Re: java       David  
99       frontend for   Hanley 
         lisp?                 
10/11/   Re: Announcing Raymond
99       CLORB a Commo  Wiker  
11/04/   Re: LispWorks  Robert 
99       status         Monfera
"""

I.e., if I understand correctly, because nnweb doesn't read
articles before that is actually requested by the user, it doesn't
know message-id and references fields and thus orders the articles
of the "Announcing CLORB.." thread randomly (well, probably in the
order it gets them or something), which in this case is the reverse
of the correct order.  Right?

I wouldn't probably care about this if Gnus didn't put the initial
article of the thread in the middle of the replies, since now that
it does, I can't immediately see which one is the initial article
(which I consider as useful information when searching for
information from articles).  But even if it doesn't have the normal
information for threading, couldn't it a) put the article with no
"Re:" in its subject as the first one and b) order the rest based
on the date.  As far as I can see, in this case this simple logic
would produce the correct ordering.  Of course, it would fail to
order replies sent on the same day, but that's not so important
anyway.  Am I making any sense at all?

-- 
Hannu

Re: Unable to create Dejanews nnweb groups

[Lars Magne Ingebrigtsen]

Hannu Koivisto <azure@iki.fi> writes:

> I.e., if I understand correctly, because nnweb doesn't read
> articles before that is actually requested by the user, it doesn't
> know message-id and references fields and thus orders the articles
> of the "Announcing CLORB.." thread randomly (well, probably in the
> order it gets them or something), which in this case is the reverse
> of the correct order.  Right?

I've looked the code over, and it looks like it retains the order that 
it gets from Deja.

It uses the following parameters for the search:

    (nnweb-encode-www-form-urlencoded
     `(("ST" . "PS")
       ("svcclass" . "dnyr")
       ("QRY" . ,search)
       ("defaultOp" . "AND")
       ("DBS" . "1")
       ("OP" . "dnquery.xp")
       ("LNG" . "ALL")
       ("maxhits" . "100")
       ("threaded" . "0")
       ("format" . "verbose2")
       ("showsort" . "date")
       ("agesign" . "1")
       ("ageweight" . "1")))))

Perhaps the agesign should be something else?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen