zsh-workers
 help / color / mirror / code / Atom feed
* out of memory error after "edit-command-line"
@ 2022-08-02  3:47 Jim
  2022-08-12 15:33 ` Jim
  0 siblings, 1 reply; 4+ messages in thread
From: Jim @ 2022-08-02  3:47 UTC (permalink / raw)
  To: devs


[-- Attachment #1.1: Type: text/plain, Size: 1165 bytes --]

Hi everyone,

ZSH_VERSION: 5.8 (Distribution, Devuan, still hasn't update)
$ZSH_PATCHLEVEL:  debian/5.8-6+deb11u1
default editor nvim version v0.4.4
Also tested with VIM - Vi IMproved 8.2
terminal emulators/multiplexer:  tmux, xfce4-terminal, st, and
kitty(checked on all)

.zshrc
  autoload -Uz edit-command-line
  zle -N edit-command-line
  bindkey -M vicmd v edit-command-line

Created an anonymous function(see attachment ps_via_zsh) by entering
edit-command-line
which seems to work ok after exiting editor and hitting enter. But
then(after adding code) started
getting out of memory errors(see attachment out_of_memory).  When I recall
the anonymous
function from history and execute, no error. Started commenting out lines
and found that the issues
stops with the line:

CL=${(0A)$(</proc/$P/cmdline)} ; PP=${AA[PPid]} ; Cmd=${AA[Name]}

commented out. Splitting line found issue to be with
CL=${(0A)$(</proc/$P/cmdline)}
Again, recalling from history -- no edit-command-line -- no error.
BTW: /proc/<pid>/cmdline uses NULL as the separator.

QUESTION:  Bug, or am I doing something stupid? Or something with nvim and
vim?

Thanks for listening,

Jim Murphy

[-- Attachment #1.2: Type: text/html, Size: 1549 bytes --]

[-- Attachment #2: ps_via_zsh --]
[-- Type: application/octet-stream, Size: 850 bytes --]

()
{ # Z-Shell version of 'ps -o pid=,ppid=,tty=,comm=,args= $$'
  emulate -L zsh
  # FunCTioN:  ps_via_zsh
  [[ -v modules[zsh/datetime] ]] || zmodload zsh/datetime
  local ST ET
  ST=$EPOCHREALTIME
  ps -o pid=,ppid=,tty=,comm=,args= $$
  ET=$EPOCHREALTIME
  print $(( ST - ET ))
  ST=$EPOCHREALTIME
  [[ -v modules[zsh/stat] ]] || zmodload zsh/stat
  local     CL Cmd E P PP T
  local -a  A B
  local -A  AA
  P=$$
  T="$(zstat +link /proc/$P/fd/0)"
  A=(${(M)${${(f)"$(</proc/$P/status)"}:s/:/,}:#(Name|PPid),*})
  for E ($A) {
    B=(${(s.,.)E})
    B[2]=${(MS)B[2]##[[:graph:]]*[[:graph:]]} # remove leading/trailing white space
    AA[$B[1]]="$B[2]"
  }
  CL=${(0A)$(</proc/$P/cmdline)} ; PP=${AA[PPid]} ; Cmd=${AA[Name]}
  print -- ${(l.5.. .)P} ${(l.5.. .)PP} ${(r.12.. .)T} ${(r.15.. .)Cmd} $CL
  ET=$EPOCHREALTIME
  print $(( ST - ET ))
}

[-- Attachment #3: out_of_memory --]
[-- Type: application/octet-stream, Size: 1347 bytes --]

[ 2065.764363] Out of memory: Killed process 12153 (zsh) total-vm:6812316kB, anon-rss:6802224kB, file-rss:0kB, shmem-rss:0kB, UID:17227 pgtables:13360kB oom_score_adj:0
[ 1681.950581] Out of memory: Killed process 12116 (zsh) total-vm:6818304kB, anon-rss:6802244kB, file-rss:0kB, shmem-rss:0kB, UID:17227 pgtables:13384kB oom_score_adj:0
[23829.634980] Out of memory: Killed process 12375 (zsh) total-vm:5084260kB, anon-rss:5075368kB, file-rss:0kB, shmem-rss:0kB, UID:17227 pgtables:9992kB oom_score_adj:0
[35153.665324] Out of memory: Killed process 12338 (zsh) total-vm:4940764kB, anon-rss:4931592kB, file-rss:0kB, shmem-rss:0kB, UID:17227 pgtables:9712kB oom_score_adj:0
[35777.880940] Out of memory: Killed process 15754 (zsh) total-vm:5090388kB, anon-rss:5081832kB, file-rss:0kB, shmem-rss:0kB, UID:17227 pgtables:10000kB oom_score_adj:0
[36283.820854] Out of memory: Killed process 15709 (zsh) total-vm:5067208kB, anon-rss:5058760kB, file-rss:4kB, shmem-rss:0kB, UID:17227 pgtables:9952kB oom_score_adj:0
[36533.967014] Out of memory: Killed process 30580 (zsh) total-vm:5024676kB, anon-rss:5016104kB, file-rss:4kB, shmem-rss:0kB, UID:17227 pgtables:9868kB oom_score_adj:0
[37206.318183] Out of memory: Killed process 31268 (zsh) total-vm:5024168kB, anon-rss:5012684kB, file-rss:0kB, shmem-rss:0kB, UID:17227 pgtables:9872kB oom_score_adj:0

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: out of memory error after "edit-command-line"
  2022-08-02  3:47 out of memory error after "edit-command-line" Jim
@ 2022-08-12 15:33 ` Jim
  2022-08-12 20:18   ` Bart Schaefer
  0 siblings, 1 reply; 4+ messages in thread
From: Jim @ 2022-08-12 15:33 UTC (permalink / raw)
  To: devs

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

Hi again,

Update:

Tried to isolate what causes the 'out of memory' errors and found that if
I added quotes the error stopped.
  CL=${(0A)"$(< /proc/$P/cmdline)"}  # OK
  CL=${(0A)$(< /proc/$P/cmdline)}  # out of memory error

I had another script and it worked without the quotes. Compared the two
and found the other script had the option 'extendedglob' set. After adding
the option to the script I attached in the original email the 'out of
memory'
errors stopped when there are no quotes.

Not sure what globbing has to do with this. Hopefully someone can
enlighten me. But even so, should zsh ever fail with an 'out of memory'
error? Again, it only happens after exiting 'edit-command-line'.

Regards,

Jim

On Mon, Aug 1, 2022 at 10:47 PM Jim <linux.tech.guy@gmail.com> wrote:

> Hi everyone,
>
> ZSH_VERSION: 5.8 (Distribution, Devuan, still hasn't update)
> $ZSH_PATCHLEVEL:  debian/5.8-6+deb11u1
> default editor nvim version v0.4.4
> Also tested with VIM - Vi IMproved 8.2
> terminal emulators/multiplexer:  tmux, xfce4-terminal, st, and
> kitty(checked on all)
>
> .zshrc
>   autoload -Uz edit-command-line
>   zle -N edit-command-line
>   bindkey -M vicmd v edit-command-line
>
> Created an anonymous function(see attachment ps_via_zsh) by entering
> edit-command-line
> which seems to work ok after exiting editor and hitting enter. But
> then(after adding code) started
> getting out of memory errors(see attachment out_of_memory).  When I recall
> the anonymous
> function from history and execute, no error. Started commenting out lines
> and found that the issues
> stops with the line:
>
> CL=${(0A)$(</proc/$P/cmdline)} ; PP=${AA[PPid]} ; Cmd=${AA[Name]}
>
> commented out. Splitting line found issue to be with
> CL=${(0A)$(</proc/$P/cmdline)}
> Again, recalling from history -- no edit-command-line -- no error.
> BTW: /proc/<pid>/cmdline uses NULL as the separator.
>
> QUESTION:  Bug, or am I doing something stupid? Or something with nvim and
> vim?
>
> Thanks for listening,
>
> Jim Murphy
>

[-- Attachment #2: Type: text/html, Size: 2921 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: out of memory error after "edit-command-line"
  2022-08-12 15:33 ` Jim
@ 2022-08-12 20:18   ` Bart Schaefer
  2022-08-13  0:10     ` [PATCH] " Bart Schaefer
  0 siblings, 1 reply; 4+ messages in thread
From: Bart Schaefer @ 2022-08-12 20:18 UTC (permalink / raw)
  To: linuxtechguy; +Cc: devs

On Fri, Aug 12, 2022 at 8:41 AM Jim <linux.tech.guy@gmail.com> wrote:
>
> Tried to isolate what causes the 'out of memory' errors and found that if
> I added quotes the error stopped.
>   CL=${(0A)"$(< /proc/$P/cmdline)"}  # OK
>   CL=${(0A)$(< /proc/$P/cmdline)}  # out of memory error

I didn't reproduce the out-of-memory error with zsh-5.9-31, but I did
reproduce an infinite loop in wordcount() because skipwsep() does not
consider a metafied NUL byte to be whitespace.  spacesplit() is
entered with e.g. 'zsh\203 -f\203 ' and calls wordcount() on the first
\203, which returns 2, then calls skipwsep() which returns without
skipping anything because iwsep(' '^32) is false, and around we go
again.

I think the OOM is related because spacesplit() will allocate memory
for each word that it believes it found, and the loop causes it to
keep finding empty words forever.  With sufficient RAM it just takes
so long to use it all up that I give up waiting and kill the shell
with gdb, which is how I found the loop case.

However, I can't come up with a minimal test case to invoke the
initial condition.  It's not enough just to do e.g.
${(0A)$(</proc/$$/cmdline)} in isolation.

Quotes matter because readoutput() does not call spacesplit() when
$(...) is quoted, but I don't know why extendedglob would make any
difference.

> But even so, should zsh ever fail with an 'out of memory' error?

There's really no way for the shell to know that it is imminently
going to run out of memory and no useful way to recover once it has
done so.  Except in cases of bugs (which this appears to be), OOM only
results from a programming error on the part of the user.

Back to the original problem ... it appears that findsep(&s, NULL, 0)
considers '\203 ' to be a separator (because a null byte is in $IFS ?)
but skipwsep() does not.  Can anyone else see why this doesn't break
in every case of splitting strings with embedded NUL ?


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH] Re: out of memory error after "edit-command-line"
  2022-08-12 20:18   ` Bart Schaefer
@ 2022-08-13  0:10     ` Bart Schaefer
  0 siblings, 0 replies; 4+ messages in thread
From: Bart Schaefer @ 2022-08-13  0:10 UTC (permalink / raw)
  To: linuxtechguy; +Cc: devs

[-- Attachment #1: Type: text/plain, Size: 1811 bytes --]

On Fri, Aug 12, 2022 at 1:18 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> However, I can't come up with a minimal test case to invoke the
> initial condition.  It's not enough just to do e.g.
> ${(0A)$(</proc/$$/cmdline)} in isolation.
> [...] it appears that findsep(&s, NULL, 0)
> considers '\203 ' to be a separator (because a null byte is in $IFS ?)
> but skipwsep() does not.

OK, that's a red herring.  The real problem is that itype_end(s, ISEP,
1) is not skipping over the '\203 ' pair in Jim's example, whereas it
does in the simple example above.

Once this becomes broken, it remains broken -- the simple example
starts infinite-looping as well.

It seems to come down to this in itype_end():
4359                case ISEP:
4360                if (!wmemchr(ifs_wide.chars, wc, ifs_wide.len))
4361                    return (char *)ptr;

On entry to that case in the simple example, wmemchr() returns nonzero
for wc == 0.  After edit-command-line, wmemchr() starts returning zero
in that case.  It appears ifs_wide has been erased.

The problem starts here in edit-command-line:

  # Compute the cursor's position in bytes, not characters.
  setopt localoptions nomultibyte noksharrays

When nomultibyte is set, inittyptab() is called and erases ifs_wide.
This is not restored when emulation mode ends and multibyte is
re-asserted.

The following patch fixes this example, but might only be a partial
fix for problems with locally flipping the state of various options
(MONITOR, BANGHIST, SHINSTDIN come to mind).  I think really we should
be looping over the options and calling dosetopt() for each one
instead of just memcpy'ing the saved set back on top of the original
... or at least we need a mapping of the subset of options that have
extra code associated with a change via setopt.

[-- Attachment #2: localnomultibyte.txt --]
[-- Type: text/plain, Size: 790 bytes --]

diff --git a/Src/exec.c b/Src/exec.c
index f2911807c..77763f536 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -5962,10 +5962,17 @@ doshfunc(Shfunc shfunc, LinkList doshargs, int noreturnval)
 	    sticky = funcsave->sticky;
 	} else if (isset(LOCALOPTIONS)) {
 	    /* restore all shell options except PRIVILEGED and RESTRICTED */
+#ifdef MULTIBYTE_SUPPORT
+	    int flip_multibyte = opts[MULTIBYTE];
+#endif
 	    funcsave->opts[PRIVILEGED] = opts[PRIVILEGED];
 	    funcsave->opts[RESTRICTED] = opts[RESTRICTED];
 	    memcpy(opts, funcsave->opts, sizeof(opts));
 	    emulation = funcsave->emulation;
+#ifdef MULTIBYTE_SUPPORT
+	    if (flip_multibyte != opts[MULTIBYTE])
+		inittyptab();
+#endif
 	} else {
 	    /* just restore a couple. */
 	    opts[XTRACE] = funcsave->opts[XTRACE];


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-08-13  0:21 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-02  3:47 out of memory error after "edit-command-line" Jim
2022-08-12 15:33 ` Jim
2022-08-12 20:18   ` Bart Schaefer
2022-08-13  0:10     ` [PATCH] " Bart Schaefer

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).