Re: [PATCH 1/2] x86_64/memset: avoid multiply insn if possible

[Rich Felker <dalias@libc.org>]

On Thu, Feb 12, 2015 at 06:17:02PM +0100, Denys Vlasenko wrote:
> memset is very, very often called with fill=0,
> and 64-bit imul is expensive on many CPUs.
> Avoid it if fill=0.
> 
> Also avoid multiply on "short memset" codepath if possible,
> and when we do need it, use 32-bit one, which is cheaper on many CPUs.

I'd actually like to extend the "short" range up to at least 32 bytes
using two 8-byte writes for the middle, unless the savings from using
32-bit imul instead of 64-bit are sufficient to justify 4 4-byte
writes for the middle. On the cpu I tested on, the difference is 11
cycles vs 32 cycles for non-rep path versus rep path at size 32.

> Signed-off-by: Denys Vlasenko <vda.linux@googlemail.com>
> ---
>  src/string/x86_64/memset.s | 34 +++++++++++++++++++++-------------
>  1 file changed, 21 insertions(+), 13 deletions(-)
> 
> diff --git a/src/string/x86_64/memset.s b/src/string/x86_64/memset.s
> index 3cc8fcf..523caa0 100644
> --- a/src/string/x86_64/memset.s
> +++ b/src/string/x86_64/memset.s
> @@ -1,13 +1,12 @@
>  .global memset
>  .type memset,@function
>  memset:
> -	movzbl %sil,%esi
> -	mov $0x101010101010101,%rax
> -	# 64-bit imul has 3-7 cycles latency, launch early
> -	imul %rsi,%rax
> -
> +	movzbq %sil,%rax
>  	cmp $16,%rdx
> -	jb 1f
> +	jb .Less_than_16
> +	test %esi,%esi
> +	jnz .L_widen_rax  # unlikely
> +.L_widened:

Generally I prefer unnamed labels, but I think you've reached the
level where the code is getting confusing without names, so I'm fine
with having them here.

Once you're done with improvements to the 64-bit version, do you plan
to adapt the same principles to the 32-bit version? Or should I or
someone else work on that later?

Rich