From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.3 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 23852 invoked from network); 30 May 2022 18:40:27 -0000 Received: from zero.zsh.org (2a02:898:31:0:48:4558:7a:7368) by inbox.vuxu.org with ESMTPUTF8; 30 May 2022 18:40:27 -0000 ARC-Seal: i=1; cv=none; a=rsa-sha256; d=zsh.org; s=rsa-20210803; t=1653936027; b=eNPXTD5gXvCaLSMO1q+5sm1kBBvlWz0cdl1C8REq9cL5q0mukYqLcCGWpkiqYHLNKgVUgqWT8F +XLRXV7ow9Czdn+dvG/qlIbrDqmmDH1knFrInyS7x8/tQYS8D4FAQK+fT4af5tlVdlQMe5wvbW lk6NqWW5Y4dGnrGMstY7uOP7Pxb3qQ1ZoE1QHe9dp8FsjZ7Z7jNXerdDBkQdq9xSBMXbE/2F/9 qV5CkIwMtjILUvmVhQXd3NixfIo5Dv0sstLWsxYfsUJQTGCdPtPHZBRaPKEf4XcQNWN1QDEhQN gLZSP7zrCky6SS6iZTIQXDHOzWM+0Q72USOlwCR6V6qVuw==; ARC-Authentication-Results: i=1; zsh.org; iprev=pass (mail-ej1-f41.google.com) smtp.remote-ip=209.85.218.41; dkim=pass header.d=brasslantern-com.20210112.gappssmtp.com header.s=20210112 header.a=rsa-sha256; dmarc=none header.from=brasslantern.com; arc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=zsh.org; s=rsa-20210803; t=1653936027; bh=itjD/0Awty2ZQr33N0BYFh3ihWmH+T7iyQWiQZt8wNs=; h=List-Archive:List-Owner:List-Post:List-Unsubscribe:List-Subscribe:List-Help: List-Id:Sender:Content-Type:Cc:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:DKIM-Signature:DKIM-Signature; b=WPV2TJLVyUUnmYGKv7uL3DosySUZJGwH1NSPtQBMuNUCueZ7LhbEKfgQQ6VQoezUzlVQrg9iFg j2lKUvSyAesS0rWIPPQ1T0zEZQ153MI5bWVhmnvhj6f0fCxDtBnYO3J6xUlyXHk9vcvrPPbEVd aFNPY4EyRkLWKW5UpQUZODLTiBlLivNi28kpLEWaQi4Go4jh7Ahpt10JlQMn1wgqITOF/4Expm MlTDyK4+Dt/Cn92nFKNXpGKXAKg0Rts6VAlRmNh4KauYhb5OKmOmKJS3SY+1njegCp/bPY4ys7 yJ+7QrR6DOiSWP8VubPkPqPaid1OypKyiT9ANtZFKumADQ==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zsh.org; s=rsa-20210803; h=List-Archive:List-Owner:List-Post:List-Unsubscribe: List-Subscribe:List-Help:List-Id:Sender:Content-Type:Cc:To:Subject:Message-ID :Date:From:In-Reply-To:References:MIME-Version:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=Sp9yEQJ0FS1+UfL7R2iD14SW4zrjpNiPXLJxwELJTyY=; b=Nmw1FbzfeCLTYJA10urmxWqke4 AI6lNDX0eUC2bZovcqg2S/Q35HXlapMYGa6wLqLqwFnFSr1D1cLhHLo0Wl+eR6dbsODEocmSnrBUL 0Yuhe0v9eQqhhYGTIzA02h8rE9QuIAYwrEVLH774T9Y0RF1C0AVAH+UxxTbe305531O/DXEsgq1/S MCsn7eC+YFPyKlIng/t3V4+o154W9p93P0x8v7lhGU0Jqp3lwuXIHw8bJGEzE1OtqHPrdgMCxAZLZ LK1fN3DJPwsA0eagQV858CAS7wenqIJV/UtBQd7xalwY/V3UsN56QKqvAKXfLSWozEhXULDmGZj2p k1XkicVQ==; Received: from authenticated user by zero.zsh.org with local id 1nvkJW-0004Y8-1F; Mon, 30 May 2022 18:40:26 +0000 Authentication-Results: zsh.org; iprev=pass (mail-ej1-f41.google.com) smtp.remote-ip=209.85.218.41; dkim=pass header.d=brasslantern-com.20210112.gappssmtp.com header.s=20210112 header.a=rsa-sha256; dmarc=none header.from=brasslantern.com; arc=none Received: from mail-ej1-f41.google.com ([209.85.218.41]:34597) by zero.zsh.org with esmtps (TLS1.3:TLS_AES_128_GCM_SHA256:128) id 1nvkJE-0004ED-DD; Mon, 30 May 2022 18:40:09 +0000 Received: by mail-ej1-f41.google.com with SMTP id q21so22477916ejm.1 for ; Mon, 30 May 2022 11:40:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=brasslantern-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Sp9yEQJ0FS1+UfL7R2iD14SW4zrjpNiPXLJxwELJTyY=; b=ZcCwoD2O3kBTCyUTyviPWU4IVx9GyYBKRFQtQXEBhGNiKeDll4s/t/eKr/Roxows/B n183GqLdgnFuXdn4qaHOLthJVOANgWwW5clNIVYPjMoKKepnnjokg9/uavvA4R24aSiR 9SgfZOwVzrkzxZ11+Dhtvz56ZYsjYfFl+J+vaUcfL4UU98BbppQHWB2O9hLwR5v7GIc/ CBtwCuDvVbgH5rkoQ1N2ivMejTAt4TtRnKFgS6EUnju26Ec6j/V8uMgJcLuV6vyoA7g7 RrZYaHBumO8JxH1NjhV/6V9nfl/PSDGinaqDRYahVVVrXpZPlZuEkWvkpNv1ufQ+7BYq yliw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Sp9yEQJ0FS1+UfL7R2iD14SW4zrjpNiPXLJxwELJTyY=; b=CDrM4u0/9LBzmPsJR/Gdi9iIlIkIkpdCB5qHEIu77NXndO4e+i5OM81hWsKTP0VJ5+ 7MNS1ZHb3+kAhtox2ojNG6yLYansUpFkvIwX+65we9UnyRyEC37Qz/kWgfQVSJiCi4bG vHamUYGchrTuyttgMCPqB2UBWpc2zGq11AhiBZAMy8ouCe4KfhiBR9K/VRN3NE8RMDZD zTlxN50jJUSRRJA+ygMNFmkMEgPF/2+kFyusmnc1sGZIM3kK4r4dVndSa3febfrD5HL0 GYhqJYAflt4BcKTOEnxLQbppDcZwOlBWxXQWhSUkgldU4eB++YC7VnUfI5/5zFOZBZWf OeQA== X-Gm-Message-State: AOAM5315Xc/UqhThU+D/RSUBRgDLZkeg7BDl1YjQiMFXArzy/MQ4xLUz nbGrYBzntTa5j2LFPZqwmdpLqGI6yDzXsta9aEdxgQV9qFNKjg== X-Google-Smtp-Source: ABdhPJwQ+1HEZ57EBionxaVzOmjq2tf6Dn1Pgnf8s+lz1tfLuMoa/iHMLoeefx/znH0SAZsc9mLKx51+SrAGa6y4aSY= X-Received: by 2002:a17:907:7290:b0:6ff:1fa2:e9e3 with SMTP id dt16-20020a170907729000b006ff1fa2e9e3mr19942365ejc.345.1653936007785; Mon, 30 May 2022 11:40:07 -0700 (PDT) MIME-Version: 1.0 References: <67D8ACEC-3D46-4EC4-9AAD-C508C7097499@kba.biglobe.ne.jp> In-Reply-To: <67D8ACEC-3D46-4EC4-9AAD-C508C7097499@kba.biglobe.ne.jp> From: Bart Schaefer Date: Mon, 30 May 2022 11:39:56 -0700 Message-ID: Subject: Re: Removing subshell from zargs (see "zargs with -P intermittently failing") To: "Jun. T" Cc: Zsh hackers list Content-Type: text/plain; charset="UTF-8" X-Seq: 50309 Archived-At: X-Loop: zsh-workers@zsh.org Errors-To: zsh-workers-owner@zsh.org Precedence: list Precedence: bulk Sender: zsh-workers-request@zsh.org X-no-archive: yes List-Id: List-Help: List-Subscribe: List-Unsubscribe: List-Post: List-Owner: List-Archive: On Mon, May 30, 2022 at 9:34 AM Jun. T wrote: > > Sorry I missed this post. And I fear I will not be able to going into > any detail at least for a few days. No worries. > (3) I tried the following dirty patch for debugging: > [...] > > When the problem occurs, I get: > > % zargs -n 1 -P 19 -- {1..40} -- f; jobs > zsh: 0 0 > zsh: 1 17504 # thisjob = ignorejob Aha. Yes, this is what the original subshell + wait was intended to prevent. I think what's happened here is that "waid $j" was looking for the exit of (what is here shown as) job 0, but job 1 exited first, and zsh had to handle but disregard that exit in order to continue waiting for the job whose status was wanted. Doing a full "wait" beforehand guarantees that we can return the status of any job without having to handle intervening signals, and theoretically doesn't slow us down because we eventually have to wait for all the jobs anyway. > zsh: 8 2137 > [8] + done { "${call[@]}"; } > > maxjob = 8 here, but it should have been decremented to 1? > The only place maxjob is decremented is in freejob(), I guess. Yes, once a job is in (in this case) job table slot 8, new jobs have to use 9 and up even if the jobs in slots 0 through 7 have already exited. Once the current maxjob slot is reclaimed, all slots before it that point to exited jobs can be reclaimed too. The question is why slot 8 wasn't reclaimed until "jobs" ran. Likely related in some way to the order in which the jobs exited. Harmless, I guess, but potentially confusing enough that the subshell wrapper should be left in place? Given the behavior above, it would also appear possible to fill up the parent shell job table by having maxjobs get "stuck" at increasingly larger slots. Or else it's related to this: # These setopts are necessary for "wait" on multiple jobs to work. setopt nonotify nomonitor Those options are local to the zargs function, of course, but perhaps they should actually be inside the subshell?