Patch: 3.1.6-bart-7: Another rlimit

Patch: 3.1.6-bart-7: Another rlimit

[Bart Schaefer]

Albert Chin reports that IRIX 6.5 has

     RLIMIT_PTHREAD   The maximum number of threads (pthreads(5)) that a
                      process may create.  Functions that attempt to create
                      new threads beyond this limit will fail with the error
                      EAGAIN.

Index: Doc/Zsh/builtins.yo
===================================================================
@@ -521,6 +521,7 @@
 sitem(tt(descriptors))(Maximum value for a file descriptor.)
 sitem(tt(filesize))(Largest single file allowed.)
 sitem(tt(maxproc))(Maximum number of processes.)
+sitem(tt(maxpthreads))(Maximum number of threads per process.)
 sitem(tt(memorylocked))(Maximum amount of memory locked in RAM.)
 sitem(tt(memoryuse))(Maximum resident set size.)
 sitem(tt(resident))(Maximum resident set size.)
Index: Src/Builtins/rlimits.awk
===================================================================
@@ -33,6 +33,7 @@
 	    if (limnam == "NOFILE")  { msg[limnum] = "Ndescriptors" }
 	    if (limnam == "NPROC")   { msg[limnum] = "Nmaxproc" }
 	    if (limnam == "OFILE")   { msg[limnum] = "Ndescriptors" }
+	    if (limnam == "PTHREAD") { msg[limnum] = "Nmaxpthreads" }
 	    if (limnam == "RSS")     { msg[limnum] = "Mresident" }
 	    if (limnam == "SBSIZE")  { msg[limnum] = "Msockbufsize" }
 	    if (limnam == "STACK")   { msg[limnum] = "Mstacksize" }

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

PATCH: Re: Patch: 3.1.6-bart-7: Another rlimit

[Bart Schaefer]

On Oct 19,  5:22am, Bart Schaefer wrote:
} Subject: Patch: 3.1.6-bart-7: Another rlimit
}
}      RLIMIT_PTHREAD   The maximum number of threads (pthreads(5))

The "ulimit -a" command should report this; same for "sockbufsize".

Index: Src/Builtins/rlimits.c
===================================================================
@@ -232,6 +232,20 @@
 	    limit /= 1024;
 	break;
 # endif /* RLIMIT_AIO_MEM */
+# ifdef RLIMIT_SBSIZE
+    case RLIMIT_SBSIZE:
+	if (head)
+	    printf("socket buffer size (kb)    ");
+	if (limit != RLIM_INFINITY)
+	    limit /= 1024;
+	break;
+# endif /* RLIMIT_SBSIZE */
+# ifdef RLIMIT_PTHREAD
+    case RLIMIT_PTHREAD:
+	if (head)
+	    printf("threads per process        ");
+	break;
+# endif /* RLIMIT_PTHREAD */
     }
     /* display the limit */
     if (limit == RLIM_INFINITY)

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: PATCH: Re: Patch: 3.1.6-bart-7: Another rlimit

[Zefram]

Bart Schaefer wrote:
>}      RLIMIT_PTHREAD   The maximum number of threads (pthreads(5))
>
>The "ulimit -a" command should report this; same for "sockbufsize".

Should it?  There are several limits that ulimit can't set; I intuitively
feel that it should be reporting only the same set of limits that it
can set.

Btw, the name "maxpthreads" seems sub-optimal -- what's wrong with
"maxthreads"?

-zefram

Re: PATCH: Re: Patch: 3.1.6-bart-7: Another rlimit

[Bart Schaefer]

On Oct 20,  9:17pm, Zefram wrote:
} Subject: Re: PATCH: Re: Patch: 3.1.6-bart-7: Another rlimit
}
} Bart Schaefer wrote:
} >}      RLIMIT_PTHREAD   The maximum number of threads (pthreads(5))
} >
} >The "ulimit -a" command should report this; same for "sockbufsize".
} 
} Should it?  There are several limits that ulimit can't set; I intuitively
} feel that it should be reporting only the same set of limits that it
} can set.

I wondered about that, but it was already reporting several limits that
it can't set (address space, cached threads, AIO operations and memory)
so there didn't seem to be any point in leaving out these two.

} Btw, the name "maxpthreads" seems sub-optimal -- what's wrong with
} "maxthreads"?

I went back and forth on that.  Eventually I left in the "p" because there
are other threading systems than POSIX threads and it's conceivable that a
system might have limits for more than one.  E.g., is the "cachedthreads"
limit related to POSIX threads?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

3.0 DESTDIR

[Clint Adams]

Bart, please backport the DESTDIR stuff for inclusion in 3.0.8.

Re: 3.0 DESTDIR

[Bart Schaefer]

On Oct 23,  8:45pm, Clint Adams wrote:
} Subject: 3.0 DESTDIR
}
} Bart, please backport the DESTDIR stuff for inclusion in 3.0.8.

I'll consider it, but in retrospect I'm doubtful that it should have been
included in 3.1 in the first place.  (Yes, I know I argued in favor at
the time.)

However, I now can't see how

	make DESTDIR=/foo/bar/baz install

differs in end result from

	make
	make prefix=/foo/bar/baz/usr/local install

provided that the same options were passed to "configure" in each case.
It just seems we've scattered references to $DESTDIR in a whole lot of
files (introducing several bugs that had to be fixed later along the way)
without actually achieving anything new.

If you can show me something substantially different that DESTDIR is the
best way to handle, I might swing back the other way again.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: 3.0 DESTDIR

[Clint Adams]

> However, I now can't see how
> 
> 	make DESTDIR=/foo/bar/baz install
> 
> differs in end result from
> 
> 	make
> 	make prefix=/foo/bar/baz/usr/local install

Well, I can answer that since my surprise at those two not being effectively
the same is precisely the reason I wanted DESTDIR in the first place:
(illustration using bart7)

% ./configure --prefix=/usr
[output elided]
% make
[output elided]
% make prefix=/tmp/zshtest/usr install
[output elided]
% strings /tmp/zshtest/usr/bin/zsh | grep tmp
/tmp/zshtest/usr/share/zsh/functions
/tmp/zshtest/usr/lib/zsh/3.1.6-bart-7
/tmp/zsh
/tmp/zsh
/var/log/wtmp
/var/run/utmp
% /tmp/zshtest/usr/bin/zsh -f
% echo $fpath
/tmp/zshtest/usr/share/zsh/functions

The functions are there, as are the modules, but if you were to
chroot to /tmp/zshtest, you'd be screwed.

DESTDIR provides a safe way to guarantee against this sort of thing
because nothing depends on it but the install, whereas prefix is used
to generate fndir and libdir, which are then used in the somewhat
unexpected recompile and relink during the install target.  I hope
that that was coherent.

Re: 3.0 DESTDIR

[Bart Schaefer]

On Oct 24,  1:05am, Clint Adams wrote:
} Subject: Re: 3.0 DESTDIR
} 
} > 	make DESTDIR=/foo/bar/baz install
} > 
} > differs in end result from
} > 
} > 	make
} > 	make prefix=/foo/bar/baz/usr/local install
}
} Well, I can answer that since my surprise at those two not being effectively
} the same is precisely the reason I wanted DESTDIR in the first place:
} (illustration using bart7)
} 
} % strings /tmp/zshtest/usr/bin/zsh | grep tmp
} /tmp/zshtest/usr/share/zsh/functions
} /tmp/zshtest/usr/lib/zsh/3.1.6-bart-7

Ah, yes, I had noticed that recompile, too, but:

(1) that doesn't apply to 3.0.x, because it doesn't compile any of those
strings into the binary, so you still haven't given a reason to have the
DESTDIR patch in that version; and

(2) that recompile is a bug in the 3.1.6 build process which I think we
ought to fix whether or not DESTDIR is also available; and

(3) DESTDIR isn't sufficient anyway, because simply prefixing the entire
path from the root forces you to install into a precise mirror tree of
the install hierarchy.  It turns out that in one of the cases where I
thought DESTDIR would be useful, it actually is not; because what I want
is to change /usr/local/* into /usr/local/stow/zsh/*, that is, insert a
new hierarchy _below_ ${prefix} rather than above.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: 3.0 DESTDIR

[Clint Adams]

> (1) that doesn't apply to 3.0.x, because it doesn't compile any of those
> strings into the binary, so you still haven't given a reason to have the
> DESTDIR patch in that version; and

Fair enough.  I withdraw the request.

> (2) that recompile is a bug in the 3.1.6 build process which I think we
> ought to fix whether or not DESTDIR is also available; and

Agreed.

> (3) DESTDIR isn't sufficient anyway, because simply prefixing the entire
> path from the root forces you to install into a precise mirror tree of
> the install hierarchy.  It turns out that in one of the cases where I
> thought DESTDIR would be useful, it actually is not; because what I want
> is to change /usr/local/* into /usr/local/stow/zsh/*, that is, insert a
> new hierarchy _below_ ${prefix} rather than above.

Yes, DESTDIR is useful for automated package building and bootstrapping
a chrooted environment, but not what you describe.

However, I'm not sure why you would want to install files into
/usr/local/stow that believe their functions/modules are going to be
in /usr/local.

Re: 3.0 DESTDIR

[Bruce Stephens]

Clint Adams <schizo@debian.org> writes:

> Yes, DESTDIR is useful for automated package building and bootstrapping
> a chrooted environment, but not what you describe.

Can't you use "make install prefix=..." there, too?

PATCH: 3.1.6 install without rebuild (Re: 3.0 DESTDIR)

[Bart Schaefer]

This is a whole lot of verbiage ending in a very short patch.

On Oct 24, 10:30am, Clint Adams wrote:
} Subject: Re: 3.0 DESTDIR
}
} > (2) that recompile is a bug in the 3.1.6 build process which I think we
} > ought to fix whether or not DESTDIR is also available; and
} 
} Agreed.

The issue appears to be that the instructions for building zshpaths.h are
considered to be a module definition task (they're in zsh.mdd) rather than
a configuration task.  So they don't get pasted into a Makefile until the
"make -f Makemod" gets run, and hence are compile-time rather than config-
time substituted.  This is compounded by dependence on $(VERSION), which
is a make-time rather than config-time value as well.

However, I think the following patch will do the right things:  Before the
patch, the target zshpaths.h depends on FORCE, and then the rule avoids
touching the file if it hasn't changed.  The patch simply replaces FORCE
with a dependency on the .mk files that define the variables that become
part of the strings in zshpaths.h; so if you re-configure, or "cvs update"
to a newer version of the sources, zshpaths.h will get remade.

This means that "make MODDIR=/some/new/path fndir=/some/other/new/path"
does not recompile zsh, but "make ... install" still installs the modules
and functions in the new $(MODDIR) and $(fndir) locations.

I'm now considering backing out the $DESTDIR stuff, because with this patch
you can do this kind of thing (which works in 3.0.7 as well, by the way):

	configure --prefix='${INSTROOT}/usr/local'
	make
	make INSTROOT=/tmp/zshtest install

} However, I'm not sure why you would want to install files into
} /usr/local/stow that believe their functions/modules are going to be
} in /usr/local.

There's a GNU package called "stow" (based on something called "depot"
that originated at CMU).  It's a package-installation utility designed
for use in shared-filesystem environments.  Rather than copy the files
from the package directly into place the way an RPM or Debian package
does, it expects to find the packages in subdirectories of a directory
(usually also named "stow") whose *parent* directory is the equivalent
of the ${prefix}.  It constructs relative symlinks pointing into each
package's tree in the corresponding subdirs of the parent directory.

E.g., /usr/local/bin/zsh -> ../stow/zsh/bin/zsh after "stow zsh".

Thus the same "stow" directory can be NFS mounted on multiple machines,
and each of those machines can selectively install/uninstall packages
without consuming additional disk space.  This doesn't work for every
package, because some things get security indigestion if they find a
symlink in their execution path; but it's great for things like elisp
libraries and zsh functions that are architecture-independent and can
usually be updated independently of the interpreters that run them.

Here's the patch.

Index: Src/zsh.mdd
===================================================================
@@ -12,6 +12,8 @@
 prototypes.h hashtable.h ztype.h"
 
 :<<\Make
+@CONFIG_MK@
+
 signames.c: signames1.awk signames2.awk ../config.h @SIGNAL_H@
 	$(AWK) -f $(sdir)/signames1.awk @SIGNAL_H@ >sigtmp.c
 	$(CPP) sigtmp.c >sigtmp.out
@@ -28,7 +30,7 @@
 version.h: $(sdir_top)/Config/version.mk
 	echo '#define ZSH_VERSION "'$(VERSION)'"' > $@
 
-zshpaths.h: FORCE Makemod
+zshpaths.h: Makemod $(CONFIG_INCS)
 	@echo '#define MODULE_DIR "'$(MODDIR)'"' > zshpaths.h.tmp
 	@if test x$(fndir) != xno; then \
 	  echo '#define FPATH_DIR "'$(fndir)'"' >> zshpaths.h.tmp; \

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: PATCH: 3.1.6 install without rebuild (Re: 3.0 DESTDIR)

[Zefram]

Bart Schaefer wrote:
>This means that "make MODDIR=/some/new/path fndir=/some/other/new/path"
>does not recompile zsh,

I intentionally wrote that code so that that *would* cause a recompile.
make MODDIR=foo guarantees to build a zsh which has the module directory
specified.  After your patch, changing MODDIR for make results in a zsh
with indeterminate module directory.  You would require a "make clean"
to reliably change the module directory, when in reality only a couple of
files need to be rebuilt.  (I'm operating under a more general principle
that the final result of make should be the same regardless of files
already built; make's job is simply to minimise the amount that gets
rebuilt, reusing a partial build if possible.)

Btw, my view is that giving different parameters to "make install"
from those one gave to "make" is a bad practice, but if it is to be
supported then the parameters from "make install" must take precedence.
After all, those are the parameters that will take effect if anything
changes and gets rebuilt during the "make install".

All of this is a matter of "least surprise", too.

>I'm now considering backing out the $DESTDIR stuff, because with this patch
>you can do this kind of thing (which works in 3.0.7 as well, by the way):
>
>	configure --prefix='${INSTROOT}/usr/local'
>	make
>	make INSTROOT=/tmp/zshtest install

Surely you're joking, Mr Schaefer.  OK, that's a nice hack, but DESTDIR
is a whole lot clearer.  Let's keep configured pathnames and install
paths properly distinct.

-zefram

Re: PATCH: 3.1.6 install without rebuild (Re: 3.0 DESTDIR)

[Bart Schaefer]

On Oct 24, 10:04pm, Zefram wrote:
} Subject: Re: PATCH: 3.1.6 install without rebuild (Re: 3.0 DESTDIR)
}
} Bart Schaefer wrote:
} >This means that "make MODDIR=/some/new/path fndir=/some/other/new/path"
} >does not recompile zsh,
} 
} I intentionally wrote that code so that that *would* cause a recompile.

I understand that.

} Btw, my view is that giving different parameters to "make install"
} from those one gave to "make" is a bad practice, but if it is to be
} supported then the parameters from "make install" must take precedence.

Unfortunately, it's my understanding that the GNU-accepted practice for
changing the install location of software that is under autoconf/configure
control is exactly what my patch permits.  Of course we're not actually
GNU-ware (thank goodness) so we can ignore that if we choose, but it's
been causing me to have to copy things into the install directories by
hand for as long as I've been building 3.1.x.

Did you read my description of the "stow" utility?  If "make install"
alters the binary, how does one produce a correctly-configured binary
that can be managed by such a utility?

} After all, those are the parameters that will take effect if anything
} changes and gets rebuilt during the "make install".

Yes, that's true.  It's up to a package installer to do the steps in the
right order:  That is, never "make install" without first doing "make".

} Let's keep configured pathnames and install paths properly distinct.

Exactly.  Paths that are hardwired into compiled components should be
those specified at configure time, not those specified at "make" time.
That's why they call it "configure".

Unfortunately, without a much more significant rewrite of the module
build system, we can't achieve that.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: PATCH: 3.1.6 install without rebuild (Re: 3.0 DESTDIR)

[Bart Schaefer]

On Oct 24, 10:04pm, Zefram wrote:
} Subject: Re: PATCH: 3.1.6 install without rebuild (Re: 3.0 DESTDIR)
}
} [...]  After your patch, changing MODDIR for make results in a zsh
} with indeterminate module directory.  You would require a "make clean"
} to reliably change the module directory, when in reality only a couple of
} files need to be rebuilt.

Incidentally, "make clean" is NOT required if you rerun "configure" instead.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: 3.0 DESTDIR

[Clint Adams]

> Can't you use "make install prefix=..." there, too?

In 3.0 you can.

No in 3.1, because the function and modules directories are reset and
hardcoded to the absolute paths.

Many other programs do this and don't support DESTDIR, and they are
an annoyance.  perl for instance.  GNU awk.  texinfo.  texutils..

Re: PATCH: 3.1.6 install without rebuild (Re: 3.0 DESTDIR)

[Juergen A. Erhard]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>>>>> "Bart" == Bart Schaefer <schaefer@candle.brasslantern.com> writes:

[SNIP]

A few minor (and probably totally unimportant ;-) corrections:

    Bart> There's a GNU package called "stow" (based on something
    Bart> called "depot" that originated at CMU).

From stow.texi:

   Stow was inspired by Carnegie Mellon's Depot program, [...]

It is definitely not based on it (in the narrower sense).

    Bart> It's a package-installation utility designed for use in
    Bart> shared-filesystem environments.

Also according to stow.texi, it's for keeping files from different
packages apart, nothing more, nothing less.  So, when you have "a2p.1"
in "/usr/local/man/man1", you can easily find out which package this
file is from.

Of course you can *use* it for what you describe ;-) (And maybe that
was depot's intended use).

[SNIP]

Bye, J

PS: It's good to see support for stow... I use it for everything under
/usr/local.  Aaaand building Debian (and I guess RH) packages poses
some of the same problems: you need to `install' into a different
place than the one the package later has to run in.

- -- 
Jürgen A. Erhard      eMail: jae@ilk.de      phone: (GERMANY) 0721 27326
    MARS: http://members.tripod.com/~Juergen_Erhard/mars_index.html
            GNUstep - Free OPENSTEP (http://www.gnustep.org)
   "No matter how cynical I get, I can't keep up."  -- Bruce Schneier
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.0 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjgXrckACgkQN0B+CS56qs1ijQCfWK6UjW7RC6nswVX2ONmKKz9f
9KMAn2+nDEsej0AohXyr6RigDWsOGlvP
=Gi++
-----END PGP SIGNATURE-----

Re: PATCH: 3.1.6 install without rebuild (Re: 3.0 DESTDIR)

[Bart Schaefer]

On Oct 28,  3:58am, Juergen A. Erhard wrote:
} Subject: Re: PATCH: 3.1.6 install without rebuild (Re: 3.0 DESTDIR)
}
} Also according to stow.texi, it's for keeping files from different
} packages apart, nothing more, nothing less.  [...]
} 
} Of course you can *use* it for what you describe ;-) (And maybe that
} was depot's intended use).

I find this amusing, considering that I've sat no more than twelve feet
from Bob Glickstein almost every working day for the past seven years.
Yes, I know what he wrote in stow.texi, but trust me when I say that I
also know exactly what stow's author uses stow for!

As for whether it's "based on" depot ... the perl script was "depot.pl"
for the first four years of its life, right up until Bob decided it had
to be called something else if he was going to release it through FSF.

However, this now has absolutely nothing to do with zsh, so to make this
at least slightly topical:  I'll be offline from tomorrow through the
weekend, so with the exception of a couple of messages tonight I probably
won't be answering any zsh mail until Tuesday.  Don't pile the patches
too deep in the meantime ...

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com