How to add string to end of each array element without loops

How to add string to end of each array element without loops

[nix]

a=(1 2 3)
a=(foo$^a[@])

print -l "${(@n)a}"

foo1
foo2
foo3

I would like to get the following output instead:

1foo
2foo
3foo

Re: How to add string to end of each array element without loops

[Mikael Magnusson]

On 25 March 2011 22:32,  <nix@myproxylists.com> wrote:
> a=(1 2 3)
> a=(foo$^a[@])
>
> print -l "${(@n)a}"
>
> foo1
> foo2
> foo3
>
> I would like to get the following output instead:
>
> 1foo
> 2foo
> 3foo

a=(${^a}foo) ?

-- 
Mikael Magnusson

Re: How to add string to end of each array element without loops

[nix]

> On 25 March 2011 22:32,  <nix@myproxylists.com> wrote:
>> a=(1 2 3)
>> a=(foo$^a[@])
>>
>> print -l "${(@n)a}"
>>
>> foo1
>> foo2
>> foo3
>>
>> I would like to get the following output instead:
>>
>> 1foo
>> 2foo
>> 3foo
>
> a=(${^a}foo) ?
>
> --
> Mikael Magnusson
>

Worked, thank you. For some reason subscripts are giving to me headaches.
Maybe I should ghet more fresh air to clear my head out :)

Re: How to add string to end of each array element without loops

[Bart Schaefer]

On Mar 25, 11:32pm, nix@myproxylists.com wrote:
} Subject: How to add string to end of each array element without loops
}
} a=(1 2 3)
} a=(foo$^a[@])
} 
} print -l "${(@n)a}"

If $^a[@] works, then you don't need the subscript, either:

a=(foo$^a)

Futhermore you don't need the quotes and (@) in the print expression:

print -l ${(n)a}

} foo1
} foo2
} foo3
} 
} I would like to get the following output instead:
} 
} 1foo
} 2foo
} 3foo

I'm not understanding what you're asking.  Either this is as easy as

    a=(1 2 3)
    a=(${^a}foo)  # braces needed to distinguish $a from $afoo

or you've already got (foo$^a) and you're asking how to change "foo?"
into "?foo" for the whole array

    setopt extendedglob
    a=(${a/(#b)foo(<->)/$match[1]foo})

or you mean something else entirely.  Can you elaborate?

-- 

Re: How to add string to end of each array element without loops

[nix]

> On Mar 25, 11:32pm, nix@myproxylists.com wrote:
> } Subject: How to add string to end of each array element without loops
> }
> } a=(1 2 3)
> } a=(foo$^a[@])
> }
> } print -l "${(@n)a}"
>
> If $^a[@] works, then you don't need the subscript, either:
>
> a=(foo$^a)
>
> Futhermore you don't need the quotes and (@) in the print expression:
>
> print -l ${(n)a}
>
> } foo1
> } foo2
> } foo3
> }
> } I would like to get the following output instead:
> }
> } 1foo
> } 2foo
> } 3foo
>
> I'm not understanding what you're asking.  Either this is as easy as
>
>     a=(1 2 3)
>     a=(${^a}foo)  # braces needed to distinguish $a from $afoo
>
> or you've already got (foo$^a) and you're asking how to change "foo?"
> into "?foo" for the whole array
>
>     setopt extendedglob
>     a=(${a/(#b)foo(<->)/$match[1]foo})
>
> or you mean something else entirely.  Can you elaborate?
>
> --
>

10.0.0.1 	10.127.255.254

> On 25 March 2011 22:32,  <nix@myproxylists.com> wrote:
>> a=(1 2 3)
>> a=(foo$^a[@])
>>
>> print -l "${(@n)a}"
>>
>> foo1
>> foo2
>> foo3
>>
>> I would like to get the following output instead:
>>
>> 1foo
>> 2foo
>> 3foo
>
> a=(${^a}foo) ?
>
> --
> Mikael Magnusson
>

I wanted to append word 'foo' to end of each element in an array and
Mikael's solution did it.

Now however, im getting a segfault when an array elements exceeds over 4
million elements even there are enough RAM in my box.

Is there a way to increase this limit? There was nothing to that with
'limit' command.

Re: How to add string to end of each array element without loops

[Bart Schaefer]

On Mar 26,  3:37am, nix@myproxylists.com wrote:
}
} Now however, im getting a segfault when an array elements exceeds over
} 4 million elements even there are enough RAM in my box.

A segfault doing what, exactly?  Perhaps run with:

exec 2>/tmp/zsh$$trace
setopt xtrace

I can create and reference arrays of that size without a problem:

schaefer<523> typeset -A a
schaefer<524> a=({4000000..1} {1..4000000})
schaefer<525> echo $#a
4000000
schaefer<526> a=({5000000..1} {1..5000000})
schaefer<527> echo $#a
5000000

and then after noodling around a bit:

schaefer<533> b=($a)
schaefer<534> echo $#b
5000000

even if I guarantee there are multiple simultaneous copies of the array:

schaefer<535> foo() { local -a c; c=($b); print $#c }
schaefer<536> foo
5000000

so it's not something magic about 4 million.

} Is there a way to increase this limit? There was nothing to that with
} 'limit' command.

The limit command (and ulimit) deal with OS-imposed limitations on the
resource usage of a process.  It's entirely possible that it's one of
those that you're running up against, but difficult to guess which one
without knowing what the shell was doing at the moment of failure.

Generally speaking zsh doesn't impose any of its own restrictions except
for function recursion depth and number of simultaneous monitored jobs
(both of which are limited to 1000).

Re: How to add string to end of each array element without loops

[nix]

> On Mar 26,  3:37am, nix@myproxylists.com wrote:
> }
> } Now however, im getting a segfault when an array elements exceeds over
> } 4 million elements even there are enough RAM in my box.
>
> A segfault doing what, exactly?  Perhaps run with:
>
> exec 2>/tmp/zsh$$trace
> setopt xtrace
>
> I can create and reference arrays of that size without a problem:
>
> schaefer<523> typeset -A a
> schaefer<524> a=({4000000..1} {1..4000000})
> schaefer<525> echo $#a
> 4000000
> schaefer<526> a=({5000000..1} {1..5000000})
> schaefer<527> echo $#a
> 5000000
>
> and then after noodling around a bit:
>
> schaefer<533> b=($a)
> schaefer<534> echo $#b
> 5000000
>
> even if I guarantee there are multiple simultaneous copies of the array:
>
> schaefer<535> foo() { local -a c; c=($b); print $#c }
> schaefer<536> foo
> 5000000
>
> so it's not something magic about 4 million.
>
> } Is there a way to increase this limit? There was nothing to that with
> } 'limit' command.
>
> The limit command (and ulimit) deal with OS-imposed limitations on the
> resource usage of a process.  It's entirely possible that it's one of
> those that you're running up against, but difficult to guess which one
> without knowing what the shell was doing at the moment of failure.
>
> Generally speaking zsh doesn't impose any of its own restrictions except
> for function recursion depth and number of simultaneous monitored jobs
> (both of which are limited to 1000).
>

./SUBNET
[+] Generating IP-addresses ... please wait
[+] 2072640 IP-addresses has been generated ...
[+] Generating proxy list ... please wait
WTF ...
Segmentation fault

The code (SUBNET):

#!/bin/zsh

zmodload -i zsh/terminfo
zmodload -i zsh/datetime
zmodload -i zsh/mathfunc
zmodload -i zsh/pcre
zmodload -i zsh/mapfile

emulate zsh

exec 2>/tmp/zsh$$trace
setopt xtrace

#HOST_MIN="10.100.12.1"
#HOST_MAX="10.100.12.14"

#HOST_MIN="10.100.12.1"
#HOST_MAX="10.100.15.254"

#HOST_MIN="10.96.0.1"
#HOST_MAX="10.111.255.254"

#HOST_MIN="183.190.0.1"
#HOST_MAX="183.191.255.254"

HOST_MIN="128.96.0.1"
HOST_MAX="128.127.255.254"

#HOST_MIN="128.96.0.1"
#HOST_MAX="128.111.255.254"

IFS="."

HOST_MIN_A=$(print ${${HOST_MIN[*]}[(w)1]})
HOST_MIN_B=$(print ${${HOST_MIN[*]}[(w)2]})
HOST_MIN_C=$(print ${${HOST_MIN[*]}[(w)3]})
HOST_MIN_D=$(print ${${HOST_MIN[*]}[(w)4]})

HOST_MAX_A=$(print ${${HOST_MAX[*]}[(w)1]})
HOST_MAX_B=$(print ${${HOST_MAX[*]}[(w)2]})
HOST_MAX_C=$(print ${${HOST_MAX[*]}[(w)3]})
HOST_MAX_D=$(print ${${HOST_MAX[*]}[(w)4]})

unset IFS

typeset -gA IPS
typeset -gA PARTS

PORTS="80 1080"

Proxy_List="$PWD/PROXIES.NIX"

[ -f "$Proxy_List" ] && { rm -f $Proxy_List }

echo "[+] Generating IP-addresses ... please wait"

# 0/16 - 0/23

if [ "$HOST_MIN_A" = "$HOST_MAX_A" ] && [ "$HOST_MIN_B" = "$HOST_MAX_B" ]
&& [ "$HOST_MIN_C" != "$HOST_MAX_C" ] ; then

START="$HOST_MIN_A.$HOST_MIN_B."

let "HOST_MAX_C++"

until [ "$HOST_MIN_C" -eq "$HOST_MAX_C" ] ; do

for i in {1..254} ; do

IP="$START$HOST_MIN_C.$i"
IPS[$IP]=$IP

done

let "HOST_MIN_C++"

done

# 0/24

elif [ "$HOST_MIN_A" = "$HOST_MAX_A" ] && [ "$HOST_MIN_B" = "$HOST_MAX_B"
] && [ "$HOST_MIN_C" = "$HOST_MAX_C" ] ; then

START="$HOST_MIN_A.$HOST_MIN_B.$HOST_MIN_C."

for i in {1..254} ; do

IP="$START$i"
IPS[$IP]=$IP

done

# 0/8 - 0/15

elif [ "$HOST_MIN_A" = "$HOST_MAX_A" ] && [ "$HOST_MIN_B" != "$HOST_MAX_B"
] ; then

START="$HOST_MIN_A."

let "HOST_MAX_B++"

until [ "$HOST_MIN_B" -eq "$HOST_MAX_B" ] ; do

for i in {1..255} ; do

PART="$START$HOST_MIN_B.$i."
PARTS[$PART]=$PART

[ $i = "255" ] && { let "HOST_MIN_B++" ; break }
done

done

foreach PART ($PARTS)

for x in {1..254} ; do

IP="$PART$x"
IPS[$IP]=$IP
done

end

unset PARTS

else

echo "Error: This CIDR or IP & Subnet Mask is not currently supported." ;
exit
fi

NoIPs="${#IPS[*]}"
NoPorts=$(print ${(w)#PORTS})

Total=$[NoIPs * $NoPorts]

echo "[+] $NoIPs IP-addresses has been generated ..."
echo "[+] Generating proxy list ... please wait"

for PORT in `echo $PORTS` ; do

PROXIES=()
PROXIES+=(${^IPS}:$PORT)

echo "WTF ..."

print -l ${(n)PROXIES} >> $Proxy_List

unset PROXIES

done

echo "[+] $Total proxies has been generated. Proxy list $Proxy_List is now
ready for scanning."

---

Line 143: print -l ${(n)PROXIES} >> $Proxy_List causes segfault. I can't
see where the problem is, if you try IP range up to one million, it will
work good.

PS. Beware of about 1,45GB RAM usage while using that script.

http://myproxylists.com/online-subnet-calculator

Enter the following CIDR: 128.100.12.0/11 to get a 2097150 hosts
(elements) to 'SUBNET' script array. This info is already set into
'SUBNET' but just to get a picture what im trying to do.

As you might already figured out, PHP code does subnet calculation and my
ZSH script 'SUBNET' generates IP-ranges accordingly. Quite cool is not
after segfault bug has been fixed.

Re: How to add string to end of each array element without loops

[Bart Schaefer]

On Mar 26,  4:39am, nix@myproxylists.com wrote:
}
} Line 143: print -l ${(n)PROXIES} >> $Proxy_List causes segfault. I can't
} see where the problem is, if you try IP range up to one million, it will
} work good.

Aha.  I'm able to reproduce this with

typeset -A a
a=({5000000..1} {1..5000000})
print -l $a >> /dev/null

What's actually failing is builtin.c line 283, the declaration

	VARARR(char *, argarr, argc + 1);

This is attempting to allocate the entire argument list of "print"
(which in my example is 5 million words) on the C function call
stack.  Even with "unlimit stacksize" this is likely to overflow.
The program merely doesn't discover that this has failed until a
few lines later when it tries to use the first element of the array.

There's really nothing that can be done about this.  The whole shell
paradigm of passing arguments as an argv[] array means that $a has
to be expanded and then "print" called with the words by value.

So what you have to do to work around this is NOT rely on "print -l"
to insert the newlines, and instead insert them yourself:

print ${(F)PROXIES} >> $Proxy_List

This passes a single giant word with embedded newlines to "print" and
thus avoids allocating all that space on the C stack.

You're really past the design limits here of what a language with the
semantic rules of a command shell is meant to deal with.  If "print"
were not a builtin you'd have blown out the limits of argument passing
before even getting as far as you did; even with the (F) trick, using
/bin/echo will fail with "argument list too long".

I noticed you loaded the mapfile module even though you don't use it.
It should work to do

mapfile[$Proxy_list]=${(F)PROXIES}

but whether that's actually faster than "print >>" I haven't checked.

Re: How to add string to end of each array element without loops

[nix]

> On Mar 26,  4:39am, nix@myproxylists.com wrote:
> }
> } Line 143: print -l ${(n)PROXIES} >> $Proxy_List causes segfault. I can't
> } see where the problem is, if you try IP range up to one million, it will
> } work good.
>
> Aha.  I'm able to reproduce this with
>
> typeset -A a
> a=({5000000..1} {1..5000000})
> print -l $a >> /dev/null
>
> What's actually failing is builtin.c line 283, the declaration
>
> 	VARARR(char *, argarr, argc + 1);
>
> This is attempting to allocate the entire argument list of "print"
> (which in my example is 5 million words) on the C function call
> stack.  Even with "unlimit stacksize" this is likely to overflow.
> The program merely doesn't discover that this has failed until a
> few lines later when it tries to use the first element of the array.
>

I actually tried to double the limit for stacksize 'limit stacksize 32'
but it did not fixed it. Looks like the limit is something like one
million for 'print -l $a >> /dev/null'.

> There's really nothing that can be done about this.  The whole shell
> paradigm of passing arguments as an argv[] array means that $a has
> to be expanded and then "print" called with the words by value.
>
> So what you have to do to work around this is NOT rely on "print -l"
> to insert the newlines, and instead insert them yourself:
>
> print ${(F)PROXIES} >> $Proxy_List
>
> This passes a single giant word with embedded newlines to "print" and
> thus avoids allocating all that space on the C stack.
>
> You're really past the design limits here of what a language with the
> semantic rules of a command shell is meant to deal with.  If "print"
> were not a builtin you'd have blown out the limits of argument passing
> before even getting as far as you did; even with the (F) trick, using
> /bin/echo will fail with "argument list too long".
>

./SUBNET
[+] Generating IP-addresses ... please wait
[+] 2072640 IP-addresses has been generated ...
[+] Generating proxy list ... please wait
[+] 6217920 proxies has been generated.

The (F} trick fixed it. Im quite confident that nearly none else expect me
need that big arrays. 'SUBNET' script is very fast, it's actually able to
beat in terms of speed solarwinds's advanced subnet calculator when it
comes to generating IPs out of CIDR :)

As I said earlier, my C programming skills are very limited but im getting
quite good with ZSH and PHP. Mixing two powerful scripting languages (or
programming languages) you name it, the results are quite powerful.

> I noticed you loaded the mapfile module even though you don't use it.
> It should work to do
>
> mapfile[$Proxy_list]=${(F)PROXIES}
>
> but whether that's actually faster than "print >>" I haven't checked.
>

I load other modules as well because most of my other tools are using
those modules. I tried also with mapfile but it did segfaulted as well.
Afterwards I noticed it's not even needed in 'SUBNET' script.

Thanks a bunch again. Looks like I have found my ZSH guru :)