Compiled format specs in .newsrc.eld?

Compiled format specs in .newsrc.eld?

[Jesper Harder]

Hi!

The manual says that:

   Gnus will save these compiled specs in the `.newsrc.eld' file.

But when I look in my .newsrc.eld, I don't see any trace of byte-code.
Is it a bug?

-- 
Cheers,
Jesper Harder

Re: Compiled format specs in .newsrc.eld?

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> Hi!
>
> The manual says that:
>
   To help with this, you can run `M-x gnus-compile' after you've
fiddled around with the variables and feel that you're (kind of)
satisfied.  This will result in the new specs being byte-compiled, and
you'll get top speed again.  Gnus will save these compiled specs in the
`.newsrc.eld' file.  (User-defined functions aren't compiled by this
function, though--you should compile them yourself by sticking them
into the `.gnus.el' file and byte-compiling that file.)

>
> But when I look in my .newsrc.eld, I don't see any trace of byte-code.
> Is it a bug?

Does it appear if you run M-x gnus-compile?

Re: Compiled format specs in .newsrc.eld?

[Jesper Harder]

jas@extundo.com (Simon Josefsson) writes:

> Jesper Harder <harder@ifa.au.dk> writes:
>
>> But when I look in my .newsrc.eld, I don't see any trace of byte-code.
>> Is it a bug?
>
> Does it appear if you run M-x gnus-compile?

No. 

To test it, I 

1. Created a brand new user account (so there's no existing .newsrc.eld)
2. Started Gnus
3. Changed `gnus-summary-line-format' and `gnus-group-line-format'.
4. Ran `gnus-compile', and 
5. Quit Gnus.

When I look in .newsrc.eld, there are no byte-compiled functions.

Re: Compiled format specs in .newsrc.eld?

[Kai Großjohann]

Jesper Harder <harder@ifa.au.dk> writes:

> But when I look in my .newsrc.eld, I don't see any trace of byte-code.
> Is it a bug?

gnus-compile compiles from a string to Lisp code.  Byte-compiling the
Lisp code would be another step, which is not implemented.

kai
-- 
~/.signature is: umop 3p!sdn    (Frank Nobis)

Re: Compiled format specs in .newsrc.eld?

[Jesper Harder]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Jesper Harder <harder@ifa.au.dk> writes:
>
>> But when I look in my .newsrc.eld, I don't see any trace of byte-code.
>> Is it a bug?
>
> gnus-compile compiles from a string to Lisp code.  Byte-compiling the
> Lisp code would be another step, which is not implemented.

Hmm, are you sure?  The doc-string for `gnus-compile' is

     "Byte-compile the user-defined format specs."

and when I look at the function that seems to be exactly what it does.

I think `gnus-update-format-specifications' is the defun responsible for
converting format strings to Lisp.

But in any case, neither the non-compiled Lisp versions nor the
byte-compiled versions are saved in my .newsrc.eld.

Re: Compiled format specs in .newsrc.eld?

[Kai Großjohann]

Jesper Harder <harder@ifa.au.dk> writes:

> Hmm, are you sure?  The doc-string for `gnus-compile' is
>
>      "Byte-compile the user-defined format specs."
>
> and when I look at the function that seems to be exactly what it does.
>
> I think `gnus-update-format-specifications' is the defun responsible for
> converting format strings to Lisp.
>
> But in any case, neither the non-compiled Lisp versions nor the
> byte-compiled versions are saved in my .newsrc.eld.

Whee.  It seems you're right.  Hm.  So where does the bytecode go?
Strange.

kai
-- 
~/.signature is: umop 3p!sdn    (Frank Nobis)

Re: Compiled format specs in .newsrc.eld?

[Katsumi Yamaoka]

>>>>> In <vafn0tvr145.fsf@INBOX.auto.gnus.tok.lucy.cs.uni-dortmund.de> 
>>>>>	Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:

Kai> Whee.  It seems you're right.  Hm.  So where does the bytecode go?
Kai> Strange.

The functions gnus-clear-system and gnus-read-newsrc-file do:

(delete 'gnus-format-specs gnus-variable-list)

Maybe they need to use copy-sequence.
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: Compiled format specs in .newsrc.eld?

[Kai Großjohann]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> The functions gnus-clear-system and gnus-read-newsrc-file do:
>
> (delete 'gnus-format-specs gnus-variable-list)
>
> Maybe they need to use copy-sequence.

... or remove instead of delete ...

Is anyone willing to try this?

kai
-- 
People mountain, people sea, today no see, tomorrow see.  (from Chinese)

Re: Compiled format specs in .newsrc.eld?

[Jesper Harder]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Katsumi Yamaoka <yamaoka@jpl.org> writes:
>
>> The functions gnus-clear-system and gnus-read-newsrc-file do:
>>
>> (delete 'gnus-format-specs gnus-variable-list)
>>
>> Maybe they need to use copy-sequence.
>
> ... or remove instead of delete ...
>
> Is anyone willing to try this?

Bingo!  Yep, replacing delete with remove fixes the problem:
`gnus-format-specs' is saved in .newsrc.eld and the byte-compiled
version is saved if you run `gnus-compile'.

Re: Compiled format specs in .newsrc.eld?

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Katsumi Yamaoka <yamaoka@jpl.org> writes:
>
>> The functions gnus-clear-system and gnus-read-newsrc-file do:
>>
>> (delete 'gnus-format-specs gnus-variable-list)
>>
>> Maybe they need to use copy-sequence.
>
> ... or remove instead of delete ...
>
> Is anyone willing to try this?

I tried changing it to remove, but M-x gnus-compile still doesn't put
anything in .newsrc.eld.  There seem to be a few changes related to
"format-spec" in the ChangeLog, but I'm not sure exactly what they do.

Re: Compiled format specs in .newsrc.eld?

[Jesper Harder]

jas@extundo.com (Simon Josefsson) writes:

> I tried changing it to remove, but M-x gnus-compile still doesn't put
> anything in .newsrc.eld.

Did you quit Gnus before you checked the file?

Re: Compiled format specs in .newsrc.eld?

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> jas@extundo.com (Simon Josefsson) writes:
>
>> I tried changing it to remove, but M-x gnus-compile still doesn't put
>> anything in .newsrc.eld.
>
> Did you quit Gnus before you checked the file?

Yup, but I missed to update one place, the patch below works for me as
well.  Does someone want to commit it?

Does it improve alot to do M-x gnus-compile?  Anyone care to
benchmark?

--- gnus-start.el.~6.46.~	Fri Jun 14 17:55:20 2002
+++ gnus-start.el	Mon Jun 17 16:13:18 2002
@@ -611,7 +611,7 @@
 (defun gnus-clear-system ()
   "Clear all variables and buffers."
   ;; Clear Gnus variables.
-  (let ((variables (delete 'gnus-format-specs gnus-variable-list)))
+  (let ((variables (remove 'gnus-format-specs gnus-variable-list)))
     (while variables
       (set (car variables) nil)
       (setq variables (cdr variables))))
@@ -2021,7 +2021,7 @@
   "Read startup file.
 If FORCE is non-nil, the .newsrc file is read."
   ;; Reset variables that might be defined in the .newsrc.eld file.
-  (let ((variables (delete 'gnus-format-specs gnus-variable-list)))
+  (let ((variables (remove 'gnus-format-specs gnus-variable-list)))
     (while variables
       (set (car variables) nil)
       (setq variables (cdr variables))))

Re: Compiled format specs in .newsrc.eld?

[Jesper Harder]

jas@extundo.com (Simon Josefsson) writes:

> Does it improve alot to do M-x gnus-compile?

I don't think `gnus-compile' matters much.  The *really* important
formats specs (e.g. the one controlling summary buffer lines) are
byte-compiled automatically (unless you set `gnus-use-byte-compile' to
nil).

>  Anyone care to benchmark?

There's no difference for summary buffer generation.  Of course, that's
hardly surprising, since there *shouldn't* be any difference.

Re: Compiled format specs in .newsrc.eld?

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> jas@extundo.com (Simon Josefsson) writes:
>
>> Does it improve alot to do M-x gnus-compile?
>
> I don't think `gnus-compile' matters much.  The *really* important
> formats specs (e.g. the one controlling summary buffer lines) are
> byte-compiled automatically (unless you set `gnus-use-byte-compile' to
> nil).
>
>>  Anyone care to benchmark?
>
> There's no difference for summary buffer generation.  Of course, that's
> hardly surprising, since there *shouldn't* be any difference.

So why the we bother fixing it? ;-)

Re: Compiled format specs in .newsrc.eld?

[Jesper Harder]

jas@extundo.com (Simon Josefsson) writes:

>> There's no difference for summary buffer generation.  Of course, that's
>> hardly surprising, since there *shouldn't* be any difference.
>
> So why the we bother fixing it? ;-)

The real reason why I noticed:  Someone reported a strange Gnus bug in
dk.edb.system.unix.  When starting Gnus he would get this error:

      "Symbol's value as variable is void: gnus-t"

and nothing more would happen.  I guessed that maybe it could be caused
by a wrong newline in the format specs in .newsrc.eld¹.  But when I
looked in my own .newsrc.eld there were no format specs.

I wasn't really sure whether the manual or the code was wrong about saving
format specs -- I can think of two good reasons for not saving them:

* It doesn't really matter.
* Byte-code isn't portable between Emacs and XEmacs.

So basically I was just wondering if it was a documentation bug or a
code bug.


¹ this was indeed the problem.

Re: Compiled format specs in .newsrc.eld?

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> I wasn't really sure whether the manual or the code was wrong about saving
> format specs -- I can think of two good reasons for not saving them:
>
> * It doesn't really matter.
> * Byte-code isn't portable between Emacs and XEmacs.

Yup.  Is the manual correct below?  Does Gnus byte-compile
format-specs by default internally?  Perhaps we should remove
gnus-format-specs from gnus-variable-list completely.  If there is no
speed gain in M-x gnus-compile I vote for removing it.

,----
|    If you use lots of these advanced thingies, you'll find that Gnus
| gets quite slow.  This can be helped enormously by running `M-x
| gnus-compile' when you are satisfied with the look of your lines.
| *Note Compilation::.
| ...
| Compilation
| ===========
| 
|    Remember all those line format specification variables?
| `gnus-summary-line-format', `gnus-group-line-format', and so on.  Now,
| Gnus will of course heed whatever these variables are, but,
| unfortunately, changing them will mean a quite significant slow-down.
| (The default values of these variables have byte-compiled functions
| associated with them, while the user-generated versions do not, of
| course.)
| 
|    To help with this, you can run `M-x gnus-compile' after you've
| fiddled around with the variables and feel that you're (kind of)
| satisfied.  This will result in the new specs being byte-compiled, and
| you'll get top speed again.  Gnus will save these compiled specs in the
| `.newsrc.eld' file.  (User-defined functions aren't compiled by this
| function, though--you should compile them yourself by sticking them
| into the `.gnus.el' file and byte-compiling that file.)
| 
`----

Re: Compiled format specs in .newsrc.eld?

[Jesper Harder]

jas@extundo.com (Simon Josefsson) writes:

> Jesper Harder <harder@ifa.au.dk> writes:
>
>> I wasn't really sure whether the manual or the code was wrong about saving
>> format specs -- I can think of two good reasons for not saving them:
>>
>> * It doesn't really matter.
>> * Byte-code isn't portable between Emacs and XEmacs.
>
> Yup.  Is the manual correct below?  

I think this part is wrong:

 | This will result in the new specs being byte-compiled, and you'll
 | get top speed again.  Gnus will save these compiled specs in the
 | `.newsrc.eld' file.

It's true that if you run `gnus-compile', Gnus will save the byte
compiled values in newsrc.eld.  But if I

* Run `gnus-compile'
* Quit Gnus

then at this stage the byte-compiled values are in .newsrc.eld.  
But if I

* Start Gnus again, and quit.

then the byte-compiled values in newsrc.eld have been replaced with the
uncompiled values.  

So, it appears that if you want the specs to be byte-compiled, you need
(gnus-compile) in .gnus.  This is probably a bug -- but I don't
understand what's going on.

> Does Gnus byte-compile format-specs by default internally?

I'm not sure any more.  I was probably wrong, because I looked at the
global value of `gnus-summary-line-format-spec', but I think the buffer
local value is actually used when generating the summary-buffer.

> Perhaps we should remove gnus-format-specs from gnus-variable-list
> completely.  If there is no speed gain in M-x gnus-compile I vote for
> removing it.

I think someone else should also try to verify whether `gnus-compile'
makes a big difference.  The difference I'm measuring is the smaller
than the variance.  But I'm a bit suspicious about the results -- I
would have expected it to make a much bigger difference.

BTW, I'm using this method to profile it -- it times how long it takes
to show the oldest 10.000 messages from comp.text.tex (I don't do expiry
on the server, so they're always the same):

(defun gnus-profile ()
  (interactive)
  (let (t1 t2 no (sum 0) (avg 0))
    (gnus)
    (sit-for 1)
    (dotimes (i 5)
      (gnus-group-jump-to-group "comp.text.tex")
      (garbage-collect)
      (sit-for 1)
      (setq t1 (float-time))
      (gnus-topic-select-group -10000)
      (setq t2 (float-time))
      (add-to-list 'no (- t2 t1))
      (sit-for 1)
      (gnus-summary-exit)
      (sit-for 1))
    (gnus-group-exit)
    (switch-to-buffer (get-buffer-create "gnus-profile"))
    (dolist (i no)
      (insert (format "%f " i))
      (setq sum (+ sum i)))
    (insert (format "\nSum: %f \n" sum))
    (setq avg (/ sum (length no)))
    (insert (format "Avg: %f \n" avg))
    (insert (format "Var: %f \n" (varians no)))))

(defun varians (d)
  (let ((avg (/ (apply '+ d) (length d)))
	(tmp 0))
    (dolist (x d)
      (incf tmp (expt (- x avg) 2)))
    (sqrt (/ tmp (- (length d) 1)))))

Re: Compiled format specs in .newsrc.eld?

[Kai Großjohann]

Jesper Harder <harder@ifa.au.dk> writes:

> I don't think `gnus-compile' matters much.  The *really* important
> formats specs (e.g. the one controlling summary buffer lines) are
> byte-compiled automatically (unless you set `gnus-use-byte-compile' to
> nil).

Really?  Whee.  Then we shouldn't store the byte-compiled thingies in
.newsrc.eld, I guess!

kai
-- 
People mountain, people sea, today no see, tomorrow see.  (from Chinese)

Re: Compiled format specs in .newsrc.eld?

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Yup, but I missed to update one place, the patch below works for me as
> well.  Does someone want to commit it?

Done.

> Does it improve alot to do M-x gnus-compile?  Anyone care to
> benchmark?

I noticed a speedup some time ago, when it was working.  I will see
the change soon.

I wonder: if I put (gnus-compile) in the init file, then the
main-memory version of the variables should still be there, right?  So
the only difference is that previously the variables were computed at
each start of Gnus, and now they are read from the .newsrc.eld.  I'm
confused.

Anyhow, the change can be reverted easily enough.

kai
-- 
People mountain, people sea, today no see, tomorrow see.  (from Chinese)

Re: Compiled format specs in .newsrc.eld?

[Katsumi Yamaoka]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=iso-2022-jp-2, Size: 1090 bytes --]

>>>>> In <vaf3cvmav6u.fsf@INBOX.auto.gnus.tok.lucy.cs.uni-dortmund.de>
>>>>>	Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Gro^[.A^[N_johann) wrote:

> ... or remove instead of delete ...

>>>>> In <iluofeakn0k.fsf@latte.josefsson.org>
>>>>>	Simon Josefsson <jas@extundo.com> wrote:

> -  (let ((variables (delete 'gnus-format-specs gnus-variable-list)))
> +  (let ((variables (remove 'gnus-format-specs gnus-variable-list)))

All you are using very new Emacsen, but Emacs 20 users always
need to employ cl for it.

  remove is a compiled Lisp function in `cl-seq'.
  (remove X Y)
  not documented

I think we should put the compiler macro as follows in
dgnushack.el, or we should use the expanded version of the
macro.

(unless (featurep 'xemacs)
  (define-compiler-macro remove (&whole form item seq)
    (if (>= emacs-major-version 21)
	form
      `(delete ,item (copy-sequence ,seq)))))

(compiler-macroexpand '(remove ITEM SEQ))
 => (delete ITEM (copy-sequence SEQ))

By the way, in this particular case, we have no need to use
`delete' but `delq'. :-)
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: Compiled format specs in .newsrc.eld?

[Katsumi Yamaoka]

>>>>> In <yotl8z5dmrcm.fsf@jpl.org> 
>>>>>	Katsumi Yamaoka <yamaoka@jpl.org> wrote:

> I think we should put the compiler macro as follows in
> dgnushack.el, or we should use the expanded version of the
> macro.

> (unless (featurep 'xemacs)
>   (define-compiler-macro remove (&whole form item seq)

I've installed it.  Sorry for being forward.
-- 
Katsumi Yamaoka <yamaoka@jpl.org>