why does message-generate-headers take so long?

why does message-generate-headers take so long?

[Ted Stern]

Hi all,

Sending a message from CVS Emacs + CVS Gnus on Red Hat Enterprise
Linux Workstation 4 (x86_64) takes quite a bit of time, so I profiled
all message-* functions to see where the time hog was.  I'm inserting
just the top 9 results:

,----[ elp-results ]
| Function Name                                   Call Count  Elapsed Time  Average Time
| ==============================================  ==========  ============  ============
| message-send-and-exit                           1           20.467676     20.467676
| message-send                                    1           20.467588     20.467588
| message-send-via-mail                           1           20.24165      20.24165
| message-send-mail                               1           20.241641     20.241641
| message-generate-headers                        2           20.131608     10.065804
| message-send-mail-with-sendmail                 1           0.1094629999  0.1094629999
| message-tab                                     1           0.076816      0.076816
| message-expand-name                             1           0.076688      0.076688
| message-mail                                    1           0.069009      0.069009
`----

Clearly, the most time is being spent in message-generate-headers, and
the other long times are accumulated with it.  There are a lot of
(while (re-search-forward)) loops, but this message was just a few
lines long!

Can anyone think of a reason why that routine would take so long?  Why
is it being called twice?  Is there any way to speed it up?

Ted
-- 
 dodecatheon at gmail dot com
 Frango ut patefaciam -- I break so that I may reveal

Re: why does message-generate-headers take so long?

[Andreas Seltenreich]

Ted Stern <dodecatheon@gmail.com> writes:

> Can anyone think of a reason why that routine would take so long?

Hashcash.

> Why is it being called twice?

IIRC message conses up a scratch message buffer, but hashcash
generation is inhibited for this one.

> Is there any way to speed it up?

--8<---------------cut here---------------start------------->8---
message-generate-hashcash is a variable defined in `message.el'.
Its value is t

Documentation:
*Whether to generate X-Hashcash: headers.
You must have the "hashcash" binary installed, see `hashcash-path'.

You can customize this variable.
--8<---------------cut here---------------end--------------->8---

regards,
andreas

Re: why does message-generate-headers take so long?

[Simon Josefsson]

Andreas Seltenreich <andreas+ding@gate450.dyndns.org> writes:

> Ted Stern <dodecatheon@gmail.com> writes:
>
>> Can anyone think of a reason why that routine would take so long?
>
> Hashcash.
>
>> Why is it being called twice?
>
> IIRC message conses up a scratch message buffer, but hashcash
> generation is inhibited for this one.
>
>> Is there any way to speed it up?
>
> message-generate-hashcash is a variable defined in `message.el'.
> Its value is t

I wonder whether it is a good idea to have this enabled by default
with a long mint time as 20.  Maybe we could drop the default to, say,
18 or 17?  This will still support hashcash by default, but at the
same time don't cause noticeable delays for users.  Once people start
to appreciate hashcash, and discover that the default really is too
small, they can adjust the value manually and be prepared for the
delay.

Does anyone know if SpamAssassin assign scores for hashcash cookies
based on the mint strength?  I assume that it does.  On the other
hand, it may add a sufficient large score just for the presence of a
hashcash header to make our approach work anyway.

Thoughts?  Opinions?

Re: why does message-generate-headers take so long?

[Wes Hardaker]

>>>>> "SJ" == Simon Josefsson <jas@extundo.com> writes:

SJ> I wonder whether it is a good idea to have this enabled by default
SJ> with a long mint time as 20.  Maybe we could drop the default to, say,
SJ> 18 or 17?
...
SJ> Does anyone know if SpamAssassin assign scores for hashcash cookies
SJ> based on the mint strength?

It does, but not for strengths less than 20 it turns out.  The minimum
strength value it hits for is 20 and has incremental steps between 20
and 26, and everything 26 and over is treated as "high".  The default scores
in my install are:

score HASHCASH_20 -0.500
score HASHCASH_21 -0.700
score HASHCASH_22 -1.000
score HASHCASH_23 -2.000
score HASHCASH_24 -3.000
score HASHCASH_25 -4.000
score HASHCASH_HIGH -5.000

-- 
"In the bathtub of history the truth is harder to hold than the soap,
 and much more difficult to find."  -- Terry Pratchett

Re: why does message-generate-headers take so long?

[Simon Josefsson]

Wes Hardaker <wes@hardakers.net> writes:

>>>>>> "SJ" == Simon Josefsson <jas@extundo.com> writes:
>
> SJ> I wonder whether it is a good idea to have this enabled by default
> SJ> with a long mint time as 20.  Maybe we could drop the default to, say,
> SJ> 18 or 17?
> ...
> SJ> Does anyone know if SpamAssassin assign scores for hashcash cookies
> SJ> based on the mint strength?
>
> It does, but not for strengths less than 20 it turns out.  The minimum
> strength value it hits for is 20 and has incremental steps between 20
> and 26, and everything 26 and over is treated as "high".  The default scores
> in my install are:
>
> score HASHCASH_20 -0.500
> score HASHCASH_21 -0.700
> score HASHCASH_22 -1.000
> score HASHCASH_23 -2.000
> score HASHCASH_24 -3.000
> score HASHCASH_25 -4.000
> score HASHCASH_HIGH -5.000

Maybe the default of 20 makes sense, then.

Re: why does message-generate-headers take so long?

[gdt]

[-- Attachment #1: Type: text/plain, Size: 519 bytes --]


With async payments, the pain of hashcash is very low.  And, this only
affects people with hashcash installed on their systems.  So, I think
it's reasonable to keep it.  What would be nice (I don't have the clue
level and my papers aren't in order...) would be to be able to rerun
the async payment processsing (that starts when a reply is created)
whenever there are addresses in the header (perhaps starting a few
seconds after the cursor is not in a header line).

-- 
	Greg Troxel <gdt@work.lexort.com>

[-- Attachment #2: Type: application/pgp-signature, Size: 185 bytes --]

Re: why does message-generate-headers take so long?

[Elias Oltmanns]

Ted Stern <dodecatheon@gmail.com> wrote:
> Hi all,
>
> Sending a message from CVS Emacs + CVS Gnus on Red Hat Enterprise
> Linux Workstation 4 (x86_64) takes quite a bit of time, so I profiled
> all message-* functions to see where the time hog was.
[...]
> Clearly, the most time is being spent in message-generate-headers, and
> the other long times are accumulated with it.  There are a lot of
> (while (re-search-forward)) loops, but this message was just a few
> lines long!

The search functions in cvs emacs that eventually call search_buffer
are really slow if the search string contains an i. This is because
the dotless i is now considered equivalent to the dotted i in
case-insensitive searches. This dotless i is a unicode character which
prevents search_buffer from using the fast boyer moore algorithm (if I
remember the name correctly).

I've raised this issue on emacs-devel quite some time ago but
apparently nobody cared to fix it since. Please see
<http://permalink.gmane.org/gmane.emacs.devel/53901>
for further information.

My temporary workaround is the following snippet in ~/.emacs:

--8<---------------cut here---------------start------------->8---
(unless (< emacs-major-version 22)
  (set-case-syntax 331856 "w" (standard-case-table))
  (set-case-syntax 331857 "w" (standard-case-table)))
--8<---------------cut here---------------end--------------->8---

Hope that helps.

Elias