From: sl@9front.org
To: 9front@9front.org
Subject: rc hangs forever concatenating a huge list
Date: Wed, 23 Jul 2014 02:00:35 -0400 [thread overview]
Message-ID: <43a5cf905fffdc046dea091ef7af4108@x230.inri> (raw)
Yesterday, mischief tried to upload a rather large amount of text to okturing.com,
which seemed to hang forever (no error, but the upload just continued to upload,
forever, until interrupted). We were both able to reproduce the hang reliably simply
by uploading large amounts of text. Small uploads succeeded as normal.
After poking around a bit I determined that the fault probably lies with okturing's
CGI, which attempts to store that huge input in a single list, and then later attempts
to concatenate that list into a single string.
Here is a test example that should be reproducible on any system:
; cd /lib/rfc
; wc rfc-index.xml
229468 695874 9953256 rfc-index.xml
; tail -1 rfc-index.xml
</rfc-index>
; a=`{cat rfc-index.xml}
; echo $#a
695874
; echo $a($#a)
</rfc-index>
As you can see, the entire contents of the file is accessible from list $a. No
problem so far.
However, when we attempt to concatenate the list as a string:
; echo $"a
This hangs indefinitely. Additionally, the Del key now seems to have no effect.
Interestingly, the echo command is absent from the output of ps. So, maybe rc is
choking while trying to process its arguments and has not yet attempted to run the
echo.
While hung, the rc process in question will fall into a state similar to this:
; ps | grep 339327
sl 339327 4:11 0:01 191808K Running rc
; cat /proc/339327/fd
/lib/rfc
0 r M 79379 (0000000000000001 0 00) 8192 74 /mnt/term/dev/cons
1 w M 79379 (0000000000000001 0 00) 8192 258 /mnt/term/dev/cons
2 w M 79379 (0000000000000001 0 00) 8192 258 /mnt/term/dev/cons
3 r M 79363 (00000000000080dd 2 00) 8192 512 /rc/lib/rcmain
4 r M 79379 (0000000000000001 0 00) 8192 74 /mnt/term/dev/cons
5 w M 79379 (0000000000000004 0 00) 8192 8 /dev/wdir
; acid 339327
/proc/339327/text:amd64 plan 9 executable
/sys/lib/acid/port
/sys/lib/acid/amd64
acid: lstk()
strcat(p2=0xa733158)+0x1c /sys/src/libc/amd64/strcat.s:22
Xqdol()+0x122 /sys/src/cmd/rc/exec.c:672
s=0x428b68
a=0xaf4df08
n=0x428b680077d1f6
p=0xa733118
main(argv=0x7ffffeffef98,argc=0x1)+0x3d2 /sys/src/cmd/rc/exec.c:184
rcmain=0x400dd8
num=0x373233393333
bootstrap=0x2
i=0x400dd800000000
_main+0x40 /sys/src/libc/amd64/main9.s:15
acid:
I experimented with concatenating incrementally smaller $a lists and discovered that
past a certain threshold, the echo does eventually succeed, but only after a very
long delay. This suggests that the "hangs indefinitely" observed above may in fact
just be an extended period spent processing the concatenation.
Yesterday, mischief ran [unspecified profiling] on a similar test and concluded:
mischief → Xqdol()+0x122 /sys/src/cmd/rc/exec.c:672
mischief → it counts the total length of each word in the list
mischief → allocates that
mischief → and then concatenates each word together
mischief → my poor man's profiler indicates thats where the time is spent
mischief → it spends a shit ton of time just doing the strcat'ing
There seems to be no problem managing the huge list. But squashing the huge list
into a string spins our wheels.
What gives?
sl
next reply other threads:[~2014-07-23 6:00 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-07-23 6:00 sl [this message]
2014-07-23 13:29 ` [9front] " cinap_lenrek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=43a5cf905fffdc046dea091ef7af4108@x230.inri \
--to=sl@9front.org \
--cc=9front@9front.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).