Total expiry not working -- is clock skew the problem?

Total expiry not working -- is clock skew the problem?

[Ted Stern]

Hi,

I started a new job about 100 days ago and had to reconstitute my
entire Gnus setup, almost from scratch.  In my old job, I had total
expiry turned on and articles expired with no problems.  Now I have
total expiry turned on in the group parameters for an entire topic,
but nothing has expired automatically since day one.

For example, here are the group parameters for my 'lists' topic:

      ;;; Editing the topic parameters for `lists'.
      ;; Type `C-c C-c' after you've finished editing.
      
      ((gcc-self . t)
       (total-expire . t)
       (expiry-wait . 7.0)
       (display . 100))

And here are the group parameters for a group under that topic:

      ;;; Editing the group parameters for `nnfolder:lists.modules'.
      ;; Type `C-c C-c' after you've finished editing.
      
      ((timestamp 17384 51314)
       (to-address . "modules-interest@lists.sourceforge.net")
       (to-list . "modules-interest@lists.sourceforge.net"))

But no joy.  I still have the very first article in that group.

My current theory is this:  my home directory is NFS-mounted from a
remote server, and time-synching is a real problem -- none of the
clocks match.  I get "clock-skew" errors all the time when trying to
build packages, for example.  Could the clock skew be breaking expiry?

I'm due to get a new computer eventually (only 6 months after starting
:-/) and it will have space for my home directory.  But do I have to
wait for an accurate clock before expiry will start working?

Ted
-- 
 dodecatheon at gmail dot com
 Frango ut patefaciam -- I break so that I may reveal

Re: Total expiry not working -- is clock skew the problem?

[Ted Stern]

One more note -- if I set expiry-wait to 0.0 on a topic, that works
perfectly.  It is only non-zero values that do not work.

-- Ted

On  7 Feb 2006 at 08:28 UTC-0800, Ted Stern wrote:
> Hi,
>
> I started a new job about 100 days ago and had to reconstitute my
> entire Gnus setup, almost from scratch.  In my old job, I had total
> expiry turned on and articles expired with no problems.  Now I have
> total expiry turned on in the group parameters for an entire topic,
> but nothing has expired automatically since day one.
>
> For example, here are the group parameters for my 'lists' topic:
>
>       ;;; Editing the topic parameters for `lists'.
>       ;; Type `C-c C-c' after you've finished editing.
>       
>       ((gcc-self . t)
>        (total-expire . t)
>        (expiry-wait . 7.0)
>        (display . 100))
>
> And here are the group parameters for a group under that topic:
>
>       ;;; Editing the group parameters for `nnfolder:lists.modules'.
>       ;; Type `C-c C-c' after you've finished editing.
>       
>       ((timestamp 17384 51314)
>        (to-address . "modules-interest@lists.sourceforge.net")
>        (to-list . "modules-interest@lists.sourceforge.net"))
>
> But no joy.  I still have the very first article in that group.
>
> My current theory is this:  my home directory is NFS-mounted from a
> remote server, and time-synching is a real problem -- none of the
> clocks match.  I get "clock-skew" errors all the time when trying to
> build packages, for example.  Could the clock skew be breaking expiry?
>
> I'm due to get a new computer eventually (only 6 months after starting
> :-/) and it will have space for my home directory.  But do I have to
> wait for an accurate clock before expiry will start working?
>
> Ted

-- 
 dodecatheon at gmail dot com
 Frango ut patefaciam -- I break so that I may reveal

Re: Total expiry not working -- is clock skew the problem?

[Katsumi Yamaoka]

>>>>> In <lspy80nozf3.fsf@drip.ca.boeing.com> Ted Stern wrote:

> One more note -- if I set expiry-wait to 0.0 on a topic, that works
> perfectly.  It is only non-zero values that do not work.

> On  7 Feb 2006 at 08:28 UTC-0800, Ted Stern wrote:

>> I started a new job about 100 days ago and had to reconstitute my
>> entire Gnus setup, almost from scratch.  In my old job, I had total
>> expiry turned on and articles expired with no problems.  Now I have
>> total expiry turned on in the group parameters for an entire topic,
>> but nothing has expired automatically since day one.

Well, you're talking about nnfolder groups, aren't you?  If so,
how does the X-Gnus-Article-Number header in old articles look?
(You may need to type `t' in the summary buffer.)  For example,
it should be something like:

X-Gnus-Article-Number: 1234   Tue Feb  7 08:28:55 2006

Where 1234 is an article number and the rest is a timestamp.
Gnus will look at this timestamp to decide whether to expire the
article, so if there's no such header, the time stamp is
corrupted or it shows a future date, expiring will not take
place.  I have no more idea if those are going well.

BTW, expiring might be executed in your machine eight hours
earlier than the time specified with expiry-wait.  It is because
the timestamp in the X-Gnus-Article-Number header is made by the
current-time-string function and doesn't have a timezone info.
For instance:

(progn
  (setenv "TZ" "PST8PDT") ;; -0800
  (/ (time-to-seconds (time-since (current-time-string))) 3600))
 => 8.000075615555556

Re: Total expiry not working -- is clock skew the problem?

[Ted Stern]

On  7 Feb 2006 at 19:37 UTC-0800, Katsumi Yamaoka wrote:
>>>>> In <lspy80nozf3.fsf@drip.ca.boeing.com> Ted Stern wrote:
>
>> One more note -- if I set expiry-wait to 0.0 on a topic, that works
>> perfectly.  It is only non-zero values that do not work.
>
>> On  7 Feb 2006 at 08:28 UTC-0800, Ted Stern wrote:
>
>>> I started a new job about 100 days ago and had to reconstitute my
>>> entire Gnus setup, almost from scratch.  In my old job, I had total
>>> expiry turned on and articles expired with no problems.  Now I have
>>> total expiry turned on in the group parameters for an entire topic,
>>> but nothing has expired automatically since day one.
>
> Well, you're talking about nnfolder groups, aren't you?

Yes, sorry for not being more specific!

> If so, how does the X-Gnus-Article-Number header in old articles
> look?  (You may need to type `t' in the summary buffer.)

Here is one example:

   X-Gnus-Article-Number: 1   Mon Nov  7 12:45:27 2005

This looks like it should be in the right format, right?

Other details:

- I have threads sorted by date, not number.  From ~/.gnus:

  (setq gnus-sort-gathered-threads-function 'gnus-thread-sort-by-date)
  (setq gnus-thread-sort-functions
      '(
        (not gnus-thread-sort-by-most-recent-date)
        ))

- My mailing list email comes to me through a tortuous path:

    => gmail account, adding extra headers at beginning
    => fwd'd to company mail 
    => goes through company mail filter that filters for spam and adds
       extra headers
    => procmail, in which I filter out gmail forwarding headers at
       beginning.

So is it possible that Gnus is not seeing the X-Gnus-Article-Number
header because it comes after a bunch of other headers?

> For example, it should be something like:
>
> X-Gnus-Article-Number: 1234   Tue Feb  7 08:28:55 2006
>
> Where 1234 is an article number and the rest is a timestamp.  Gnus
> will look at this timestamp to decide whether to expire the article,
> so if there's no such header, the time stamp is corrupted or it
> shows a future date, expiring will not take place.  I have no more
> idea if those are going well.
>
> BTW, expiring might be executed in your machine eight hours earlier
> than the time specified with expiry-wait.  It is because the
> timestamp in the X-Gnus-Article-Number header is made by the
> current-time-string function and doesn't have a timezone info.  For
> instance:
>
> (progn
>   (setenv "TZ" "PST8PDT") ;; -0800
>   (/ (time-to-seconds (time-since (current-time-string))) 3600))
>  => 8.000075615555556

Aha.  This could be the problem.  There is a problem with time-since
input in my Emacs.

      (progn
        (setenv "TZ" "PST8PDT") ;; -0800
          (time-since (current-time-string)))

,----[ lisp debugger output ]
| Debugger entered--Lisp error: (wrong-type-argument listp "Wed Feb  8 12:45:24 2006")
|   time-since("Wed Feb  8 12:45:24 2006")
|   (progn (setenv "TZ" "PST8PDT") (time-since (current-time-string)))
|   eval((progn (setenv "TZ" "PST8PDT") (time-since (current-time-string))))
|   eval-last-sexp-1(t)
|   eval-last-sexp(t)
|   eval-print-last-sexp()
|   call-interactively(eval-print-last-sexp)
|   recursive-edit()
|   byte-code("Æ\x10	@Ç=ƒ!

Re: Total expiry not working -- is clock skew the problem?

[Ted Stern]

On  7 Feb 2006 at 19:37 UTC-0800, Katsumi Yamaoka wrote:
>>>>> In <lspy80nozf3.fsf@drip.ca.boeing.com> Ted Stern wrote:
>
>> One more note -- if I set expiry-wait to 0.0 on a topic, that works
>> perfectly.  It is only non-zero values that do not work.
>
>> On  7 Feb 2006 at 08:28 UTC-0800, Ted Stern wrote:
>
>>> I started a new job about 100 days ago and had to reconstitute my
>>> entire Gnus setup, almost from scratch.  In my old job, I had total
>>> expiry turned on and articles expired with no problems.  Now I have
>>> total expiry turned on in the group parameters for an entire topic,
>>> but nothing has expired automatically since day one.
>
> Well, you're talking about nnfolder groups, aren't you?

Yes, sorry for not being more specific!

> If so, how does the X-Gnus-Article-Number header in old articles
> look?  (You may need to type `t' in the summary buffer.)

Here is one example:

   X-Gnus-Article-Number: 1   Mon Nov  7 12:45:27 2005

This looks like it should be in the right format, right?

Other details:

- I have threads sorted by date, not number.  From ~/.gnus:

  (setq gnus-sort-gathered-threads-function 'gnus-thread-sort-by-date)
  (setq gnus-thread-sort-functions
      '(
        (not gnus-thread-sort-by-most-recent-date)
        ))

- My mailing list email comes to me through a tortuous path:

    => gmail account, adding extra headers at beginning
    => fwd'd to company mail 
    => goes through company mail filter that filters for spam and adds
       extra headers
    => procmail, in which I filter out gmail forwarding headers at
       beginning.

So is it possible that Gnus is not seeing the X-Gnus-Article-Number
header because it comes after a bunch of other headers?

> For example, it should be something like:
>
> X-Gnus-Article-Number: 1234   Tue Feb  7 08:28:55 2006
>
> Where 1234 is an article number and the rest is a timestamp.  Gnus
> will look at this timestamp to decide whether to expire the article,
> so if there's no such header, the time stamp is corrupted or it
> shows a future date, expiring will not take place.  I have no more
> idea if those are going well.
>
> BTW, expiring might be executed in your machine eight hours earlier
> than the time specified with expiry-wait.  It is because the
> timestamp in the X-Gnus-Article-Number header is made by the
> current-time-string function and doesn't have a timezone info.  For
> instance:
>
> (progn
>   (setenv "TZ" "PST8PDT") ;; -0800
>   (/ (time-to-seconds (time-since (current-time-string))) 3600))
>  => 8.000075615555556

Aha.  This could be the problem.  There is a problem with time-since
input in my Emacs.

      (progn
        (setenv "TZ" "PST8PDT") ;; -0800
          (time-since (current-time-string)))

,----[ lisp debugger output, bytecode deleted and linebreaks added ]
| Debugger entered--Lisp error: (wrong-type-argument listp \
|   "Wed Feb  8 12:45:24 2006")
|   time-since("Wed Feb  8 12:45:24 2006")
|   (progn (setenv "TZ" "PST8PDT") (time-since (current-time-string)))
|   eval((progn (setenv "TZ" "PST8PDT") (time-since (current-time-string))))
|   eval-last-sexp-1(t)
|   eval-last-sexp(t)
|   eval-print-last-sexp()
|   call-interactively(eval-print-last-sexp)
|   recursive-edit()
|   byte-code("<weird byte code deleted so posting works>" \
|   [unread-command-char debugger-args x debugger-buffer noninteractive \
|   debugger-batch-max-lines -1 debug backtrace-debug 4 t backtrace-frame \
|   lambda 5 pop-to-buffer debugger-mode debugger-setup-buffer count-lines 2 \
|   "...\n" message "%s" buffer-string kill-emacs "" nil "" \
|   recursive-edit middlestart buffer-read-only standard-output] 4)
`----

Did you mean to use 'current-time' instead of 'current-time-string'?
In any case, I don't get any GMT/PST8PDT difference.

Ted
-- 
 dodecatheon at gmail dot com
 Frango ut patefaciam -- I break so that I may reveal

Re: Total expiry not working -- is clock skew the problem?

[Katsumi Yamaoka]

>>>>> In <lspoe1hh8ze.fsf@drip.ca.boeing.com> Ted Stern wrote:

> On  7 Feb 2006 at 19:37 UTC-0800, Katsumi Yamaoka wrote:

>> (progn
>>   (setenv "TZ" "PST8PDT") ;; -0800
>>   (/ (time-to-seconds (time-since (current-time-string))) 3600))
>>  => 8.000075615555556

> Aha.  This could be the problem.  There is a problem with time-since
> input in my Emacs.

>       (progn
>         (setenv "TZ" "PST8PDT") ;; -0800
>           (time-since (current-time-string)))

> ,----[ lisp debugger output ]
>| Debugger entered--Lisp error: (wrong-type-argument listp "Wed Feb  8 12:45:24 2006")
>|   time-since("Wed Feb  8 12:45:24 2006")
>|   (progn (setenv "TZ" "PST8PDT") (time-since (current-time-string)))

That's exactly the cause of your problem.  Nnfolder will never
expire articles if that function doesn't work.  You seem to use
`time-since' defined in some Lisp module other than time-date.el.
For example, it is make-regexp.el:

http://cvs.sourceforge.net/viewcvs.py/tei/tei-emacs/tei-emacs/Attic/make-regexp.el?rev=1.2

I could reproduce the same error with it.  If you're using such
one, you can replace it with another one which doesn't contain
time-* functions.  For instance:

https://svn.r-project.org/ESS/trunk/lisp/make-regexp.el

Other possible solutions include:

1. Don't use it.
2. Load time-date.el explicitly.
3. Putting the following form in your ~/.emacs file (or
   ~/.gnus.el file) might do the trick.

(defadvice time-since (around reload-time-date_el activate)
  "Reload time-date.el if this function seems to have been overridden."
  (condition-case nil
      ad-do-it
    (error
     (load "time-date")
     ad-do-it)))

>> BTW, expiring might be executed in your machine eight hours earlier
>> than the time specified with expiry-wait.  It is because the
>> timestamp in the X-Gnus-Article-Number header is made by the
>> current-time-string function and doesn't have a timezone info.  For
>> instance:

I've modified nnfolder.el so as to use `message-make-date' which
generates a timezone info rather than `current-time-string' when
generating X-Gnus-Article-Number header.

Re: Total expiry not working -- is clock skew the problem?

[Ted Stern]

On  8 Feb 2006 at 17:11 UTC-0800, Katsumi Yamaoka wrote:
> That's exactly the cause of your problem.  Nnfolder will never
> expire articles if that function doesn't work.  You seem to use
> `time-since' defined in some Lisp module other than time-date.el.
> For example, it is make-regexp.el:
>
> http://cvs.sourceforge.net/viewcvs.py/tei/tei-emacs/tei-emacs/Attic/make-regexp.el?rev=1.2
>
> I could reproduce the same error with it.

Bingo!  You pegged it.

> If you're using such one, you can replace it with another one which
> doesn't contain time-* functions.  For instance:
>
> https://svn.r-project.org/ESS/trunk/lisp/make-regexp.el

Okay, I snagged this one.
>
> Other possible solutions include:
>
> 1. Don't use it.

Not practical in the near term.

> 2. Load time-date.el explicitly.
> 3. Putting the following form in your ~/.emacs file (or
>    ~/.gnus.el file) might do the trick.
>
> (defadvice time-since (around reload-time-date_el activate)
>   "Reload time-date.el if this function seems to have been overridden."
>   (condition-case nil
>       ad-do-it
>     (error
>      (load "time-date")
>      ad-do-it)))

I did this for good measure (in ~/.gnus), just before the call to
make-regexp :-)

>
>>> BTW, expiring might be executed in your machine eight hours earlier
>>> than the time specified with expiry-wait.  It is because the
>>> timestamp in the X-Gnus-Article-Number header is made by the
>>> current-time-string function and doesn't have a timezone info.  For
>>> instance:
>
> I've modified nnfolder.el so as to use `message-make-date' which
> generates a timezone info rather than `current-time-string' when
> generating X-Gnus-Article-Number header.

Excellent, another good fix!  I'll update my CVS gnus also.

Ted Stern
-- 
 dodecatheon at gmail dot com
 Frango ut patefaciam -- I break so that I may reveal