Quicker exit and re-enter of large groups

Quicker exit and re-enter of large groups

[Kai Grossjohann]

Hi,

I sometimes find that I've got large groups where summary buffer
generation takes quite some time.  WIBNI something like the following
was done?

- New function for quitting a newsgroup which does (switch-to-buffer
  "*Group*"), mostly.  ==> Old summary buffer is retained, reenter of
  group is fast.  Add a keybinding for this.  This is simple.  Does it
  already exist?  I think "Q" does something different?

- When I hit M-g in a group and there are new articles is it possible
  to just add them to the summary buffer without regenerating all of
  it?

kai
-- 
I wonder why nobody don't like me,
or is it de fact dat I'm ugly? -- Harry Belafonte

Re: Quicker exit and re-enter of large groups

[Wesley.Hardaker]

>>>>> "Kai" == Kai Grossjohann <grossjohann@charly.informatik.uni-dortmund.de> writes:

Kai> - New function for quitting a newsgroup which does (switch-to-buffer
Kai>   "*Group*"), mostly.  ==> Old summary buffer is retained, reenter of
Kai>   group is fast.  Add a keybinding for this.  This is simple.  Does it
Kai>   already exist?  I think "Q" does something different?

Or create a new variable called 'leave-summary-if-size-is-over' = 400
or something...  But thats a bit silly...

Kai> - When I hit M-g in a group and there are new articles is it possible
Kai>   to just add them to the summary buffer without regenerating all of
Kai>   it?

I don't know that you'd get any speed improvement there if you did any
sort of sorting (by score, for instance), as it can't just stick them
at the end...

Re: Quicker exit and re-enter of large groups

[Kai Grossjohann]

>>>>> Wesley Hardaker writes:

  Kai> - When I hit M-g in a group and there are new articles is it
  Kai> possible to just add them to the summary buffer without
  Kai> regenerating all of it?

  Wesley> I don't know that you'd get any speed improvement there if
  Wesley> you did any sort of sorting (by score, for instance), as it
  Wesley> can't just stick them at the end...

Well, suppose there's just one new message, then surely it can be
inserted in $O(n)$ time, whereas sorting all messages needs $O(n \log
n)$ time, no?

The main time killer (at least for me, I sort messages by number and
use threading but no other sorting -- no scoring or suchlike) is
summary buffer generation which takes the most time.  Thus, having to
generate only a few lines and just sticking them in at the right
positions would be a huge improvement.  I'm talking about groups with
several thousands of (ticked) messages.

Whatcha all think?
kai "can I still have an Xmas wish or do I have to wait a year?"
-- 
I wonder why nobody don't like me,
or is it de fact dat I'm ugly? -- Harry Belafonte

Re: Quicker exit and re-enter of large groups

[Ken Raeburn]

Kai Grossjohann <grossjohann@charly.informatik.uni-dortmund.de> writes:

> Well, suppose there's just one new message, then surely it can be
> inserted in $O(n)$ time, whereas sorting all messages needs $O(n \log
> n)$ time, no?

You might need to merge and/or reorder threads, though.

> The main time killer (at least for me, I sort messages by number and
> use threading but no other sorting -- no scoring or suchlike) is
> summary buffer generation which takes the most time.  Thus, having to
> generate only a few lines and just sticking them in at the right
> positions would be a huge improvement.  I'm talking about groups with
> several thousands of (ticked) messages.

I do likewise.  Last time I did some profiling, gnus-dd-mmm took a big
chunk of that time -- nearly a third.  It's called for every message.
Can we speed it up somehow?  Or, more to the point, cut the cost of a
large number of invocations, perhaps with some sort of caching?

I've still got code around for caching "fuzzy simplified" subjects to
speed up kill-same-subject and such operations, by doing caching the
first time the operation is done on an article, increasing its cost a
bit, and bypassing all the regexp code when the cache is already
primed, greatly cutting the cost on later uses.  My current version
operates by adding one field to the mail-header structure for Gnus to
store information, and a new obarray for each group.

Perhaps we could extend that to store parsed-time info for each
article, and preserve the info across two accesses to the group?
Rebuilding the summary buffer would still take time, but less than it
does now (at least, after the first time for the group).

(Lars had said long ago he'd include this code or something like it in
Red Gnus, but I think it fell off the tail end of his to-do list.  Or
maybe I get a low score these days because of rants about multi-
threaded emacs and stuff. :-)

If we had a .overview.el file, maybe we could cache some of this data
long-term for nnml....

Re: Quicker exit and re-enter of large groups

[David Moore]

Ken Raeburn <raeburn@cygnus.com> writes:
> I do likewise.  Last time I did some profiling, gnus-dd-mmm took a big
> chunk of that time -- nearly a third.  It's called for every message.
> Can we speed it up somehow?  Or, more to the point, cut the cost of a
> large number of invocations, perhaps with some sort of caching?

	You have '%d' in your gnus-summary-line-format which is your
gnus-dd-mmm cost.  You could always take it out. ;-)


	More seriously, here's a semi-package I wrote a while back to do
generic caching on pure functions (ie ones whose result only depends on
their arguments).  I had meant to clean it up and rewrite it to be a bit
more general for supporting even situations where you don't have pure
functions but the other variables don't change very often.  Hmm, and
this version uses defadvice, which I had planned to change before I
tried to convince Lars to slap it into Gnus around various things.

In your .gnus stick in something like:
(require 'cachize)
(cachize 'gnus-dd-mmm 10000)
(cachize 'mail-extract-address-components 5000)

	If you use bbdb to mark known posters, definitely use the second
one, it'll speed up summary display by an order of magnitude on large
groups.

	Note the cache is LRU, and each slot in it takes 20 bytes.  But
if you make the number of slots too small, you can thrash on the cache.
Of course, this'll make it have slighly worse performance. :)


;;; David Moore <dmoore@ucsd.edu>
;;; This uncommented caching code is in the public domain.  Enjoy.
;;;
;;; Usage:
;;;
;;; (require 'cachize)
;;; (cachize 'mail-extract-address-components 5000)
;;;
;;; Yep, that's right, no operations to resize the cache or flush them, etc.
;;; You can use ad-unadvise to turn off the caching on a function.
;;;
;;; Note: be sure to only use this on pure functions.


(defvar cachize-data (make-hashtable 100 'eq))
(defvar cachize-info nil)		; overbound with let
(defmacro cachize-info-get-size ()
  `(aref cachize-info 0))

(defmacro cachize-info-get-cache ()
  `(aref cachize-info 1))

(defmacro cachize-info-get-cnt ()
  `(aref cachize-info 2))

(defmacro cachize-info-inc-cnt ()
  `(aset cachize-info 2 (1+ (aref cachize-info 2))))

(defmacro cachize-info-get-head ()
  `(aref cachize-info 3))

(defmacro cachize-info-set-head (head)
  `(aset cachize-info 3 ,head))

(defmacro cachize-info-get-tail ()
  `(aref cachize-info 4))

(defun cachize-pop (entry)
  (let ((next (aref entry 1))
	(prev (aref entry 2)))
  (when prev
    (aset prev 1 next))
  (when next
    (aset next 2 prev))
  (aset entry 1 nil)
  (aset entry 2 nil)))

(defun cachize-insert (entry)
  (let ((head (cachize-info-get-head)))
    (aset entry 1 (aref head 1))
    (aset entry 2 head)
    (aset head 1 entry)
    (aset (aref entry 1) 2 entry)))

(defun cachize (funsym size)
  (unless (gethash funsym cachize-data)
    (let* ((head (vector nil nil nil nil))
	   (tail (vector nil nil nil nil))
	   (info (vector size
			 (make-hashtable size 'equal)
			 0
			 head
			 tail)))
      (aset head 1 tail)
      (aset tail 2 head)
      (puthash funsym info cachize-data))
    (eval `(defadvice ,funsym (around cachize activate compile)
	     "This function is slow and therefore being cachized!"
	     (let* ((cachize-info (gethash (quote ,funsym) cachize-data))
		    (cachize-cache (cachize-info-get-cache)))
	       (when (>= (cachize-info-get-cnt) (cachize-info-get-size))
		 (let ((old (aref (cachize-info-get-tail) 2)))
		   (cachize-pop old)
		   (remhash (aref old 3) cachize-cache)))
	       (let ((cachize-entry (gethash (ad-get-args 0) cachize-cache)))
		 (unless cachize-entry
		   (cachize-info-inc-cnt)
		   (let ((args (copy-tree (ad-get-args 0))))
		     (setq cachize-entry (vector nil nil nil args))
		     (puthash args cachize-entry cachize-cache))
		   ad-do-it
		   (aset cachize-entry 0 ad-return-value))
		 (cachize-pop cachize-entry)
		 (cachize-insert cachize-entry)
		 (setq ad-return-value (aref cachize-entry 0))))))))
					     

(provide 'cachize)


-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    |

Re: Quicker exit and re-enter of large groups

[Lars Magne Ingebrigtsen]

Kai Grossjohann <grossjohann@ls6.informatik.uni-dortmund.de> writes:

> - New function for quitting a newsgroup which does (switch-to-buffer
>   "*Group*"), mostly.  ==> Old summary buffer is retained, reenter of
>   group is fast.  Add a keybinding for this.  This is simple.  Does it
>   already exist?  I think "Q" does something different?

Yes.  It would be nice with such a function, though, so I've added it
to the todo list.

> - When I hit M-g in a group and there are new articles is it possible
>   to just add them to the summary buffer without regenerating all of
>   it?

Uhm...  Well, it's certainly possible, but...  On the todo list is a
command for pulling in newly arrived articles into the current
newsgroup, which isn't very difficult.  This would be the same, but I
don't know whether it's all that useful.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Quicker exit and re-enter of large groups

[Hunter Kelly]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

> Uhm...  Well, it's certainly possible, but...  On the todo list is a
> command for pulling in newly arrived articles into the current
> newsgroup, which isn't very difficult.  This would be the same, but I
> don't know whether it's all that useful.

I personally think it would be pretty dang useful, especially for mail
groups.

> -- 
> (domestic pets only, the antidote for overdose, milk.)
>   larsi@ifi.uio.no * Lars Ingebrigtsen

Hunter

Re: Quicker exit and re-enter of large groups

[Sudeep Kumar Palat]

>>>>> "Hunter" == Hunter Kelly <retnuh@corona.pixar.com> writes:

    Hunter> Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
    >> Uhm...  Well, it's certainly possible, but...  On the todo list
    >> is a command for pulling in newly arrived articles into the
    >> current newsgroup, which isn't very difficult.  This would be
    >> the same, but I don't know whether it's all that useful.

    Hunter> I personally think it would be pretty dang useful,
    Hunter> especially for mail groups.

Yup, that would be very useful.

sudeep