[emacs-w3m:08473] nnrss should borrow nnshibmun's RSS date processor...or something

[emacs-w3m:08473] nnrss should borrow nnshibmun's RSS date processor...or something

[Mark Plaksin]

Some RSS feeds provide the date in ISO 8601 date format.  sb-rss.el from
nnshimbun converts from ISO 8601 to a format that Gnus can handle.  nnrss
does no conversion so Gnus ends up saying the date is the start of the Unix
epoch.

It would be nice if the conversion function from sb-rss.el were put into
some library that nnrss could easily call.  It's not clear to me where the
best place for that would be.  For the moment, I hacked my copy of nnrss.el
to call an un-shimbuned version of shimbun-rss-process-date.

[emacs-w3m:08474] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Katsumi Yamaoka]

>>>>> In [emacs-w3m : No.08473] Mark Plaksin wrote:

> Some RSS feeds provide the date in ISO 8601 date format.  sb-rss.el from
> nnshimbun converts from ISO 8601 to a format that Gnus can handle.  nnrss
> does no conversion so Gnus ends up saying the date is the start of the Unix
> epoch.

Gnus will be able to handle ISO 8601 date if we replace every
`parse-time-string' that Gnus uses with `date-to-time' which
uses timezone.el.  However, it is effective only in Emacs 22,
because timezone.el distributed with Emacs 21 doesn't understand
ISO 8601 date.  For instance:

(timezone-parse-date "2005-12-22T13:14:03+09:00")
;; Emacs 22.0.50
 => ["2005" "12" "22" "13:14:03" nil]
;; Emacs 21.4
 => ["0" "0" "0" "0" nil]

> It would be nice if the conversion function from sb-rss.el were put into
> some library that nnrss could easily call.  It's not clear to me where the
> best place for that would be.

I think it should be done in nnrss.el since parse-time.el and
time-date.el belong to Emacs, not Gnus.

> For the moment, I hacked my copy of nnrss.el to call an un-shimbuned
> version of shimbun-rss-process-date.

It seems to have to be used before using `message-make-date' in
the `nnrss-check-group' function.  Could you present it?

BTW, why do you prefer RFC822 date rather than ISO 8601 date?
If it is for the bugfix, we should apply it to both the Gnus
trunk and the v5-10 branch.

Re: [emacs-w3m:08474] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Mark Plaksin]

[-- Attachment #1: Type: text/plain, Size: 2682 bytes --]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> In [emacs-w3m : No.08473] Mark Plaksin wrote:
>
>> Some RSS feeds provide the date in ISO 8601 date format.  sb-rss.el from
>> nnshimbun converts from ISO 8601 to a format that Gnus can handle.  nnrss
>> does no conversion so Gnus ends up saying the date is the start of the Unix
>> epoch.
>
> Gnus will be able to handle ISO 8601 date if we replace every
> `parse-time-string' that Gnus uses with `date-to-time' which
> uses timezone.el.  However, it is effective only in Emacs 22,
> because timezone.el distributed with Emacs 21 doesn't understand
> ISO 8601 date.  For instance:
>
> (timezone-parse-date "2005-12-22T13:14:03+09:00")
> ;; Emacs 22.0.50
>  => ["2005" "12" "22" "13:14:03" nil]
> ;; Emacs 21.4
>  => ["0" "0" "0" "0" nil]

The relevant timezone.el differences between 21.4 and 22.0.50 are very
small--just three regexps in timezone-parse-date.  Perhaps Gnus should
switch to date-to-time, include the newer timezone-parse-date and invoke
it when running in Emacs < 22.

Here's one way to include the newer timezone-parse-date but it looks ugly
and the Elisp docs say "In general, well-designed Lisp programs should not
use this feature [eval-after-load]."

(if (< emacs-major-version 22)
    (eval-after-load "timezone" 
      '(defun timezone-parse-date (date)
         ... ;; include definition from Emacs 22
         )))

Is there a better way or a better plan?

>> It would be nice if the conversion function from sb-rss.el were put into
>> some library that nnrss could easily call.  It's not clear to me where the
>> best place for that would be.
>
> I think it should be done in nnrss.el since parse-time.el and
> time-date.el belong to Emacs, not Gnus.

If Gnus switches to date-to-time, then no change will be needed in nnrss,
yes?

>> For the moment, I hacked my copy of nnrss.el to call an un-shimbuned
>> version of shimbun-rss-process-date.
>
> It seems to have to be used before using `message-make-date' in
> the `nnrss-check-group' function.  Could you present it?

I copied shimbun-rss-process-date, changed the first line from this:

(luna-define-method shimbun-rss-process-date ((shimbun shimbun-rss) date)

to this:

(defun map-shimbun-rss-date (date)

and then patched nnrss.el with the attached patch.

> BTW, why do you prefer RFC822 date rather than ISO 8601 date?
> If it is for the bugfix, we should apply it to both the Gnus
> trunk and the v5-10 branch.

I don't know enough to prefer either format :)  I was just trying to find a
way to make nnrss show the correct date all the time.  Making nnrss able to
do this in both the trunk and v5-10 sounds good to me though I only use
trunk.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnrss.el patch --]
[-- Type: text/x-patch, Size: 796 bytes --]

--- nnrss.el.orig	2005-12-21 12:43:03.000000000 -0500
+++ nnrss.el	2005-12-21 22:44:04.000000000 -0500
@@ -599,9 +601,10 @@
 	(setq author (or (nnrss-node-text rss-ns 'author item)
 			 (nnrss-node-text dc-ns 'creator item)
 			 (nnrss-node-text dc-ns 'contributor item)))
-	(setq date (or (nnrss-node-text dc-ns 'date item)
-		       (nnrss-node-text rss-ns 'pubDate item)
-		       (message-make-date)))
+	(setq date (map-shimbun-rss-date
+                    (or (nnrss-node-text dc-ns 'date item)
+                        (nnrss-node-text rss-ns 'pubDate item)
+                        (message-make-date))))
 	(setq comments (nnrss-node-text rss-ns 'comments item))
 	(when (setq enclosure (cadr (assq (intern (concat rss-ns "enclosure")) item)))
 	  (let ((url (cdr (assq 'url enclosure)))

[emacs-w3m:08485] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Katsumi Yamaoka]

>>>>> In [emacs-w3m : No.08484]
>>>>>	Mark Plaksin <happy@usg.edu> wrote:

> Here's one way to include the newer timezone-parse-date but it looks ugly
> and the Elisp docs say "In general, well-designed Lisp programs should not
> use this feature [eval-after-load]."

> (if (< emacs-major-version 22)
>     (eval-after-load "timezone" 
>       '(defun timezone-parse-date (date)
>          ... ;; include definition from Emacs 22
>          )))

This is similar to the way of APEL, with which I don't fully
agree.  One of the reasons is that it only hides imperfectness
of certain Emacs versions.

> Is there a better way or a better plan?

I think the best way at the present is yours, i.e., to make
nnrss always provide RFC822 date.  However, I still don't know
the reason why we should not use ISO 8601 date in nnrss
articles.  Though we might not be getting used to seeing it in
our eyes, I don't think that is so serious.

> If Gnus switches to date-to-time, then no change will be needed in nnrss,
> yes?

It will make several date-oriented features of Gnus, e.g.,
sorting, expiration, etc., work with ISO 8601 date.  However,
I'm not sure whether all those features actually work and are
actually used.  I can only say with certainty that they will
work with RFC822 date.

> I copied shimbun-rss-process-date, changed the first line from this:

> (luna-define-method shimbun-rss-process-date ((shimbun shimbun-rss) date)

> to this:

> (defun map-shimbun-rss-date (date)

> and then patched nnrss.el with the attached patch.

Thanks.  It looks good to me, and I seem to be able to rewrite
it in nnrss.el if we cannot use the shimbun code directly.

>> BTW, why do you prefer RFC822 date rather than ISO 8601 date?
>> If it is for the bugfix, we should apply it to both the Gnus
>> trunk and the v5-10 branch.

> I don't know enough to prefer either format :)  I was just trying to find a
> way to make nnrss show the correct date all the time.  Making nnrss able to
> do this in both the trunk and v5-10 sounds good to me though I only use
> trunk.

Does ISO 8601 date actually cause you pain, or give you any
trouble?  If so, we can call it a bug and we should fix it. ;-)

[emacs-w3m:08486] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Mark Plaksin]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> I think the best way at the present is yours, i.e., to make
> nnrss always provide RFC822 date.  However, I still don't know
> the reason why we should not use ISO 8601 date in nnrss
> articles.  Though we might not be getting used to seeing it in
> our eyes, I don't think that is so serious.

My original problem was that Gnus showed some RSS articles with a date of
midnight Jan 1 1970.  I found the cause to be dc:date being in ISO 8601
format.  At that point I assumed Gnus couldn't handle ISO 8601 dates at
all.  I saw that sb-rss.el handled them and thought that was a good place
to steal code from.

This feed has ISO 8601 dates: http://del.icio.us/rss/popular/  Does it give
incorrect dates for you too?

>> If Gnus switches to date-to-time, then no change will be needed in nnrss,
>> yes?
>
> It will make several date-oriented features of Gnus, e.g.,
> sorting, expiration, etc., work with ISO 8601 date.  However,
> I'm not sure whether all those features actually work and are
> actually used.  I can only say with certainty that they will
> work with RFC822 date.

Gotcha.

>> I copied shimbun-rss-process-date, changed the first line from this:
>
>> (luna-define-method shimbun-rss-process-date ((shimbun shimbun-rss) date)
>
>> to this:
>
>> (defun map-shimbun-rss-date (date)
>
>> and then patched nnrss.el with the attached patch.
>
> Thanks.  It looks good to me, and I seem to be able to rewrite
> it in nnrss.el if we cannot use the shimbun code directly.
>
>>> BTW, why do you prefer RFC822 date rather than ISO 8601 date?
>>> If it is for the bugfix, we should apply it to both the Gnus
>>> trunk and the v5-10 branch.
>
>> I don't know enough to prefer either format :)  I was just trying to find a
>> way to make nnrss show the correct date all the time.  Making nnrss able to
>> do this in both the trunk and v5-10 sounds good to me though I only use
>> trunk.
>
> Does ISO 8601 date actually cause you pain, or give you any
> trouble?  If so, we can call it a bug and we should fix it. ;-)

As mentioned above I could live with dates in any format as long as they're
correct :)  Inertia makes me prefer RFC822 format but I could adapt.  But
the main problem is getting a date of Jan 1 1970 instead of the correct
date.

Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Katsumi Yamaoka]

>>>>> In [emacs-w3m : No.08486]
>>>>>	Mark Plaksin <happy@usg.edu> wrote:

> My original problem was that Gnus showed some RSS articles with a date of
> midnight Jan 1 1970.  I found the cause to be dc:date being in ISO 8601
> format.  At that point I assumed Gnus couldn't handle ISO 8601 dates at
> all.  I saw that sb-rss.el handled them and thought that was a good place
> to steal code from.

> This feed has ISO 8601 dates: http://del.icio.us/rss/popular/  Does it give
> incorrect dates for you too?

I promise that I'll confirm and fix it within this week.

;; I have no time today because of the housework. ;-)

Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Mark Plaksin]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> In [emacs-w3m : No.08486]
>>>>>>	Mark Plaksin <happy@usg.edu> wrote:
>
>> My original problem was that Gnus showed some RSS articles with a date of
>> midnight Jan 1 1970.  I found the cause to be dc:date being in ISO 8601
>> format.  At that point I assumed Gnus couldn't handle ISO 8601 dates at
>> all.  I saw that sb-rss.el handled them and thought that was a good place
>> to steal code from.
>
>> This feed has ISO 8601 dates: http://del.icio.us/rss/popular/  Does it give
>> incorrect dates for you too?
>
> I promise that I'll confirm and fix it within this week.
>
> ;; I have no time today because of the housework. ;-)

No rush!  I just wanted to make sure I was clear.  Thanks for all of your
work!

[emacs-w3m:08489] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Katsumi Yamaoka]

>>>>> In <87lkxwadgc.fsf@stone.tss.usg.edu> Mark Plaksin wrote:

> Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> I promise that I'll confirm and fix it within this week.

> No rush!  I just wanted to make sure I was clear.  Thanks for all of your
> work!

Now I can say there's a bug because Gnus cannot necessarily
convert ISO 8601 date into some formats using gnus-treat-date-*.
Moreover, timezone.el doesn't fully support ISO 8601 date even
in Emacs 22.  So, I've made change in nnrss.el so that it always
provides RFC822 date which Gnus should understand.  It is easier
than to make Gnus fully support ISO 8601 date.

P.S. I've also fixed the way to fill text/plain parts.

[emacs-w3m:08490] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Mark Plaksin]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> In <87lkxwadgc.fsf@stone.tss.usg.edu> Mark Plaksin wrote:
>
>> Katsumi Yamaoka <yamaoka@jpl.org> writes:
>
>>> I promise that I'll confirm and fix it within this week.
>
>> No rush!  I just wanted to make sure I was clear.  Thanks for all of your
>> work!
>
> Now I can say there's a bug because Gnus cannot necessarily
> convert ISO 8601 date into some formats using gnus-treat-date-*.
> Moreover, timezone.el doesn't fully support ISO 8601 date even
> in Emacs 22.  So, I've made change in nnrss.el so that it always
> provides RFC822 date which Gnus should understand.  It is easier
> than to make Gnus fully support ISO 8601 date.

Thanks--it works great!

> P.S. I've also fixed the way to fill text/plain parts.

Ha!  I've noticed some problems in this area but haven't tracked them down
yet.  Your changes don't seem to make a difference for me.  For example,
when I W Q in the Summary buffer this text/plain part:

http://feeds.feedburner.com/arstechnica/BAaf?m=1590
Ars Technica is at CES 2006, and this quick item will detail how our coverage
will pan out over the next few days.

gets filled to this:

http://feeds.feedburner.com/arstechnica/BAaf?m=1590
Ars Technica is at CES 2006, and this quick item will detail how our
coverage
will pan out over the next few days.

fill-column is set to 75.

[emacs-w3m:08491] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Katsumi Yamaoka]

>>>>> In <878xtug279.fsf@stone.tss.usg.edu> Mark Plaksin wrote:

> Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> P.S. I've also fixed the way to fill text/plain parts.

> Ha!  I've noticed some problems in this area but haven't tracked them down
> yet.  Your changes don't seem to make a difference for me.  For example,
> when I W Q in the Summary buffer this text/plain part:

> http://feeds.feedburner.com/arstechnica/BAaf?m=1590
> Ars Technica is at CES 2006, and this quick item will detail how our coverage
> will pan out over the next few days.

> gets filled to this:

> http://feeds.feedburner.com/arstechnica/BAaf?m=1590
> Ars Technica is at CES 2006, and this quick item will detail how our
> coverage
> will pan out over the next few days.

> fill-column is set to 75.

No, what I changed was not the `W Q' code.  Formerly, no matter
how it was long, nnrss always generated a single long line in
the text/plain part like this:

#v+
Ars Technica is at CES 2006, and this quick item will detail how our coverage will pan out over the next few days.
#v-

What I did in nnrss.el was to fold the text/plain part, binding
fill-column to 7/8 of the window width.

BTW, reading <http://feeds.feedburner.com/arstechnica/> using
nnrss gave me a hint.  Some articles contain html tags, e.g.,
<p>...</p>, <blockquote>...</blockquote>, even in the text/plain
parts (they are quoted like "&lt;p&gt;" in the xml source).
Perhaps we might need to render such contents properly.  I don't
have an idea how to do it, though.

[emacs-w3m:08492] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Mark Plaksin]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> In <878xtug279.fsf@stone.tss.usg.edu> Mark Plaksin wrote:
>
>> Katsumi Yamaoka <yamaoka@jpl.org> writes:
>
>>> P.S. I've also fixed the way to fill text/plain parts.
>
>> Ha!  I've noticed some problems in this area but haven't tracked them down
>> yet.  Your changes don't seem to make a difference for me.  For example,
>> when I W Q in the Summary buffer this text/plain part:
>
>> http://feeds.feedburner.com/arstechnica/BAaf?m=1590
>> Ars Technica is at CES 2006, and this quick item will detail how our coverage
>> will pan out over the next few days.
>
>> gets filled to this:
>
>> http://feeds.feedburner.com/arstechnica/BAaf?m=1590
>> Ars Technica is at CES 2006, and this quick item will detail how our
>> coverage
>> will pan out over the next few days.
>
>> fill-column is set to 75.
>
> No, what I changed was not the `W Q' code.  Formerly, no matter
> how it was long, nnrss always generated a single long line in
> the text/plain part like this:
> #v+
> Ars Technica is at CES 2006, and this quick item will detail how our coverage will pan out over the next few days.
> #v-
> What I did in nnrss.el was to fold the text/plain part, binding
> fill-column to 7/8 of the window width.
>
> BTW, reading <http://feeds.feedburner.com/arstechnica/> using
> nnrss gave me a hint.  Some articles contain html tags, e.g.,
> <p>...</p>, <blockquote>...</blockquote>, even in the text/plain
> parts (they are quoted like "&lt;p&gt;" in the xml source).
> Perhaps we might need to render such contents properly.  I don't
> have an idea how to do it, though.

This brings me back to wondering about the utility of text/plain parts in
RSS feed articles.  I mean the "right" way to handle <p>, <blockquote>, and
such is to render them as HTML which is what happens when you look at the
text/html part.  You never know what HTML tags are going to show up in RSS
articles so you have handle them all.

[emacs-w3m:08495] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Katsumi Yamaoka]

>>>>> In <871wzlpkbf.fsf@stone.tss.usg.edu> Mark Plaksin wrote:

> Katsumi Yamaoka <yamaoka@jpl.org> writes:

>> BTW, reading <http://feeds.feedburner.com/arstechnica/> using
>> nnrss gave me a hint.  Some articles contain html tags, e.g.,
>> <p>...</p>, <blockquote>...</blockquote>, even in the text/plain
>> parts (they are quoted like "&lt;p&gt;" in the xml source).
>> Perhaps we might need to render such contents properly.  I don't
>> have an idea how to do it, though.

> This brings me back to wondering about the utility of text/plain parts in
> RSS feed articles.

I might not be qualified for saying how text/plain parts are
useful, but

> I mean the "right" way to handle <p>, <blockquote>, and
> such is to render them as HTML which is what happens when you look at the
> text/html part.  You never know what HTML tags are going to show up in RSS
> articles so you have handle them all.

I realized it is effective to render even text/plain parts as
HTML if a user prefers to see them, and we will be able to call
it a bug that text/plain parts contain raw HTML tags.  So, I've
added the following variable (and the code).  Could you try it?

`nnrss-wash-html-in-text-plain-parts'
     Non-`nil' means that `nnrss' renders text in `text/plain' parts as
     HTML.  The function specified by the `mm-text-html-renderer'
     variable (*note Display Customization: (emacs-mime)Display
     Customization.) will be used to render text.  If it is `nil',
     which is the default, text will simply be folded.  Leave it `nil',
     if you prefer to see `text/html' parts.

[emacs-w3m:08502] Re: nnrss should borrow nnshibmun's RSS date processor...or something

[Mark Plaksin]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> I realized it is effective to render even text/plain parts as
> HTML if a user prefers to see them, and we will be able to call
> it a bug that text/plain parts contain raw HTML tags.  So, I've
> added the following variable (and the code).  Could you try it?
>
> `nnrss-wash-html-in-text-plain-parts'

It works well for me--thanks!  (Sorry for the delay--life was busy.)