[Vision for new platform] syslog, sed, cron

[Vision for new platform] syslog, sed, cron

[Daniel Cegiełka]

[-- Attachment #1: Type: text/plain, Size: 1749 bytes --]

Hi,

Rich started with basic tools for unix (noXCUse):

http://git.etalabs.net/cgi-bin/gitweb.cgi?p=noxcuse;a=summary

...and this is a good time to talk about other tools/daemons.


syslog
----------
rsyslog, syslog-ng - seems to be quite a hard choice. syslog-ng needs glib etc.

sysklogd - is an old and good, but today might be a little too outdated.

socklog - recommended by suckless.org etc., http://smarden.org/socklog/

metalog - (http://metalog.sourceforge.net/). Require PCRE, but it is a
very lightweight solution. The original release has ugly gnulib dep. I
deleted the extraneous code (gnulib) and packed everything into one
file (mlog.tar). The whole is adapted to musl. This stuff requires
further work (Makefile +-DOPTIONS)...


sed
-----
sed can be a big problem. I noticed that some programs require gnu sed
for proper installation (configure, Makefile).

minised is very interesting and worth a recommendation.

http://www.exactcode.de/site/open_source/minised/

Unfortunately, even with minised I can't build packages such as
e2fsprogs, findutils or find etc.

from my find's build.log:

checking whether gcc and cc understand -c and -o together... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether ln -s works... yes
checking for a sed that does not truncate output... configure: error:
no acceptable sed could be found in $PATH

We can prepare a patches etc... but if autotools will push gnu sed
this issue will return in the future as well as with other programs...

btw. I also ported to musl/linux bsd sed (from OpenBSD), but it has
worse compatibility with gnu sed than minised.


cron
------
I recommend ncron. Works very well with musl.
http://kain.us/nk/projects/


best regards,
Daniel

[-- Attachment #2: mlog.tar --]
[-- Type: application/x-tar, Size: 71680 bytes --]

Re: [Vision for new platform] syslog, sed, cron

[Kurt H Maier]

On Wed, Aug 22, 2012 at 07:31:49PM +0200, Daniel Cegiełka wrote:
> Rich started with basic tools for unix (noXCUse):

We have also started this with sbase:
http://hg.suckless.org/sbase

Could be that the musl guys don't like a lot of the plan9-style eprintf
stuff and our methods of recursion, but the license is very liberal and
stuff could be cherrypicked and adapted to speed development.

> sed can be a big problem. I noticed that some programs require gnu sed
> for proper installation (configure, Makefile).

autotools will always assume the whole world runs gnu; I recommend
people wanting to run autotools things should be offered a gnu-compat
package or similar that makes gnu garbage available.  I would not like
to see their 'embrace and extend' stuff infect any system built by the
musl people.

Re: [Vision for new platform] syslog, sed, cron

[Rich Felker]

On Wed, Aug 22, 2012 at 07:31:49PM +0200, Daniel Cegiełka wrote:
> Hi,
> 
> Rich started with basic tools for unix (noXCUse):
> 
> http://git.etalabs.net/cgi-bin/gitweb.cgi?p=noxcuse;a=summary
> 
> ....and this is a good time to talk about other tools/daemons.

These were not really written as a basis for a new platform. Mostly
they fall into 2 or 3 categories that might overlap:

1. Simple programming exercises to see how difficult/easy some of the
standard utilities are to implement portably on top of just the
standard library.

2. Replacements for some utilities (fold, wc) in Busybox that lack
viable Unicode support.

3. Replacements for standard utilities not covered by Busybox, and for
which the GNU implementations are hopelessly tied to glibc (iconv).

With that said, the intent of my "new platform" discussion/direction
is not to come up with a list of implementations and say "you must use
these!" Which sed, etc. you use is fairly irrelevant as long as it
gets the job done. I'm more concerned with replacing overengineered
junk that's not standardized (possibly even lacking a specification)
with simpler approaches to getting the same work done.

> syslog
> ----------
> rsyslog, syslog-ng - seems to be quite a hard choice. syslog-ng needs glib etc.

Anything that uses glib is going to crash under load, so it's not
viable for core system components. There's no way to use the vast
majority of the glib functionality without encountering abort() in
library code.

> sysklogd - is an old and good, but today might be a little too outdated.
> 
> socklog - recommended by suckless.org etc., http://smarden.org/socklog/
> 
> metalog - (http://metalog.sourceforge.net/). Require PCRE, but it is a
> very lightweight solution. The original release has ugly gnulib dep. I
> deleted the extraneous code (gnulib) and packed everything into one
> file (mlog.tar). The whole is adapted to musl. This stuff requires
> further work (Makefile +-DOPTIONS)...

I'd like to see a review of the source for the different options, for
security and robustness. Features/outward-behavior are important, but
if the core is rotten it's hard to fix... On the other hand, missing
features could be tacked onto a core that's solid.

> sed
> -----
> sed can be a big problem. I noticed that some programs require gnu sed
> for proper installation (configure, Makefile).

What GNU features do they require? Reportedly busybox sed works for
many packages...

> minised is very interesting and worth a recommendation.
> 
> http://www.exactcode.de/site/open_source/minised/

Minised is non-conformant. It does not handle multibyte characters. It
might work for making configure scripts happy, but it doesn't provide
a working sed for users who expect arbitrary characters to work. Also,
if I remember correctly its regex implementation has pathologically
bad performance in terms of big-O...

> Unfortunately, even with minised I can't build packages such as
> e2fsprogs, findutils or find etc.
> 
> from my find's build.log:
> 
> checking whether gcc and cc understand -c and -o together... yes
> checking for a BSD-compatible install... /usr/bin/install -c
> checking whether ln -s works... yes
> checking for a sed that does not truncate output... configure: error:
> no acceptable sed could be found in $PATH

What does "truncate output" mean? Are they trying to process data with
embedded nuls? Or just really long lines?

> We can prepare a patches etc... but if autotools will push gnu sed
> this issue will return in the future as well as with other programs...

It would be nice to get more packages to fix this issue...

> cron
> ------
> I recommend ncron. Works very well with musl.
> http://kain.us/nk/projects/

Yes, it's very nice and by one of my friends and musl contributors.

Rich

Re: [Vision for new platform] syslog, sed, cron

[Christian Neukirchen]

Rich Felker <dalias@aerifal.cx> writes:

> What GNU features do they require? Reportedly busybox sed works for
> many packages...

Not the Linux kernel, IIRC.

-- 
Christian Neukirchen  <chneukirchen@gmail.com>  http://chneukirchen.org

Re: [Vision for new platform] syslog, sed, cron

[Daniel Cegiełka]

2012/8/22 Rich Felker <dalias@aerifal.cx>:
> On Wed, Aug 22, 2012 at 07:31:49PM +0200, Daniel Cegiełka wrote:

>> metalog - (http://metalog.sourceforge.net/). Require PCRE, but it is a
>> very lightweight solution. The original release has ugly gnulib dep. I
>> deleted the extraneous code (gnulib) and packed everything into one
>> file (mlog.tar). The whole is adapted to musl. This stuff requires
>> further work (Makefile +-DOPTIONS)...
>
> I'd like to see a review of the source for the different options, for
> security and robustness. Features/outward-behavior are important, but
> if the core is rotten it's hard to fix... On the other hand, missing
> features could be tacked onto a core that's solid.


>> cron
>> ------
>> I recommend ncron. Works very well with musl.
>> http://kain.us/nk/projects/
>
> Yes, it's very nice and by one of my friends and musl contributors.
>
> Rich

I noticed Nicholas name in musl repo/code :) I also noticed metalog in
Nicholas ninit (init.scripts/boot).I couldn't find anything better
then metalog... so I strip a bit metalog's code...

Daniel

Re: [Vision for new platform] syslog, sed, cron

[orc]

On Wed, 22 Aug 2012 20:56:30 +0200
Christian Neukirchen <chneukirchen@gmail.com> wrote:

> Rich Felker <dalias@aerifal.cx> writes:
> 
> > What GNU features do they require? Reportedly busybox sed works for
> > many packages...
> 
> Not the Linux kernel, IIRC.
> 

I had compiled Linux (3.2.x) kernel with busybox sed and ash and not
gnu.
busybox sed is good, more compilant with gnu one.

Re: [Vision for new platform] syslog, sed, cron

[Daniel Cegiełka]

2012/8/22 Christian Neukirchen <chneukirchen@gmail.com>:
> Rich Felker <dalias@aerifal.cx> writes:
>
>> What GNU features do they require? Reportedly busybox sed works for
>> many packages...
>
> Not the Linux kernel, IIRC.

I checked.. and it looks that sed from bb works with linux kernel :)

Daniel

Re: [Vision for new platform] syslog, sed, cron

[orc]

On Wed, 22 Aug 2012 19:31:49 +0200
Daniel Cegiełka <daniel.cegielka@gmail.com> wrote:

> Hi,
> 
> Rich started with basic tools for unix (noXCUse):
> 
> http://git.etalabs.net/cgi-bin/gitweb.cgi?p=noxcuse;a=summary
> 
> ...and this is a good time to talk about other tools/daemons.
> 
> 
> syslog
> ----------
> rsyslog, syslog-ng - seems to be quite a hard choice. syslog-ng needs
> glib etc.
> 
> sysklogd - is an old and good, but today might be a little too
> outdated.
> 
> socklog - recommended by suckless.org etc.,
> http://smarden.org/socklog/
> 
> metalog - (http://metalog.sourceforge.net/). Require PCRE, but it is a
> very lightweight solution. The original release has ugly gnulib dep. I
> deleted the extraneous code (gnulib) and packed everything into one
> file (mlog.tar). The whole is adapted to musl. This stuff requires
> further work (Makefile +-DOPTIONS)...
> 
> 
> sed
> -----
> sed can be a big problem. I noticed that some programs require gnu sed
> for proper installation (configure, Makefile).
> 
> minised is very interesting and worth a recommendation.
> 
> http://www.exactcode.de/site/open_source/minised/
> 
> Unfortunately, even with minised I can't build packages such as
> e2fsprogs, findutils or find etc.
> 
> from my find's build.log:
> 
> checking whether gcc and cc understand -c and -o together... yes
> checking for a BSD-compatible install... /usr/bin/install -c
> checking whether ln -s works... yes
> checking for a sed that does not truncate output... configure: error:
> no acceptable sed could be found in $PATH
> 
> We can prepare a patches etc... but if autotools will push gnu sed
> this issue will return in the future as well as with other programs...
> 
> btw. I also ported to musl/linux bsd sed (from OpenBSD), but it has
> worse compatibility with gnu sed than minised.
> 
> 
> cron
> ------
> I recommend ncron. Works very well with musl.
> http://kain.us/nk/projects/
> 
> 
> best regards,
> Daniel

There is also Rob's toybox, it's goals is to provide light versions of
userspace tools a-la busybox, but compilant with most of POSIX
standards and Linux tools.
While busybox targets embedded, toybox targets like musl targets to be
general purpose C library. You can read more here:
http://landley.net/toybox/roadmap.html

Re: [Vision for new platform] syslog, sed, cron

[Daniel Cegiełka]

2012/8/22 orc <orc@sibserver.ru>:
> On Wed, 22 Aug 2012 19:31:49 +0200

> There is also Rob's toybox, it's goals is to provide light versions of
> userspace tools a-la busybox, but compilant with most of POSIX
> standards and Linux tools.
> While busybox targets embedded, toybox targets like musl targets to be
> general purpose C library. You can read more here:
> http://landley.net/toybox/roadmap.html

yup.. but eg. sed code:

http://landley.net/hg/toybox/file/287bca550748/toys/sed.c

There is still much to be done...
Daniel

Re: [Vision for new platform] syslog, sed, cron

[Christian Neukirchen]

Daniel Cegiełka <daniel.cegielka@gmail.com> writes:

> 2012/8/22 Christian Neukirchen <chneukirchen@gmail.com>:
>> Rich Felker <dalias@aerifal.cx> writes:
>>
>>> What GNU features do they require? Reportedly busybox sed works for
>>> many packages...
>>
>> Not the Linux kernel, IIRC.
>
> I checked.. and it looks that sed from bb works with linux kernel :)

Perhaps they fixed it then.  I remember being it the reason I ported GNU
sed to sabotage...

GNU sed is one of the most unproblematic GNU tools, btw.

> Daniel
-- 
Christian Neukirchen  <chneukirchen@gmail.com>  http://chneukirchen.org

Re: [Vision for new platform] syslog, sed, cron

[Daniel Cegiełka]

2012/8/22 Rich Felker <dalias@aerifal.cx>:
> On Wed, Aug 22, 2012 at 07:31:49PM +0200, Daniel Cegiełka wrote:

>> Unfortunately, even with minised I can't build packages such as
>> e2fsprogs, findutils or find etc.
>>
>> from my find's build.log:
>>
>> checking whether gcc and cc understand -c and -o together... yes
>> checking for a BSD-compatible install... /usr/bin/install -c
>> checking whether ln -s works... yes
>> checking for a sed that does not truncate output... configure: error:
>> no acceptable sed could be found in $PATH
>
> What does "truncate output" mean? Are they trying to process data with
> embedded nuls? Or just really long lines?

I don't know.. but I found in bb's sed code:

	/* Lie to autoconf when it starts asking stupid questions. */
	if (argv[1] && strcmp(argv[1], "--version") == 0) {
		puts("This is not GNU sed version 4.0");
		return 0;
	}

Daniel

Re: [Vision for new platform] syslog, sed, cron

[Rich Felker]

On Wed, Aug 22, 2012 at 10:08:51PM +0200, Christian Neukirchen wrote:
> Daniel Cegiełka <daniel.cegielka@gmail.com> writes:
> 
> > 2012/8/22 Christian Neukirchen <chneukirchen@gmail.com>:
> >> Rich Felker <dalias@aerifal.cx> writes:
> >>
> >>> What GNU features do they require? Reportedly busybox sed works for
> >>> many packages...
> >>
> >> Not the Linux kernel, IIRC.
> >
> > I checked.. and it looks that sed from bb works with linux kernel :)
> 
> Perhaps they fixed it then.  I remember being it the reason I ported GNU
> sed to sabotage...

I think what happened is we fixed musl. There were several
show-stopping BRE (old "basic" regex mode) bugs in TRE that made it
into musl, and sed uses BRE, not ERE.

Rich

Re: [Vision for new platform] syslog, sed, cron

[John Spencer]

On 08/23/2012 12:46 AM, Rich Felker wrote:
> On Wed, Aug 22, 2012 at 10:08:51PM +0200, Christian Neukirchen wrote:
>>
>> Perhaps they fixed it then.  I remember being it the reason I ported GNU
>> sed to sabotage...
> I think what happened is we fixed musl. There were several
> show-stopping BRE (old "basic" regex mode) bugs in TRE that made it
> into musl, and sed uses BRE, not ERE.
>
> Rich
>
it was mainly my bug hunting and dalias (musl side) & denys vlasenko's 
(bb side) bug fixing effort to get sabotage to work without gnu sed 
which led to a couple of busybox sed and musl TRE fixes between bb 
1.20.0 and 1.20.2 releases.
as of busybox 1.20.2, its sed works for every one of the ~200 packages 
in my sabotage fork (yes, even kernel and ncurses), so its seems to be 
"good" now.

Re: [Vision for new platform] syslog, sed, cron

[Jens Staal]

[-- Attachment #1: Type: text/plain, Size: 427 bytes --]

Den 22 aug 2012 22:09 skrev "Christian Neukirchen" <chneukirchen@gmail.com>:

>
> Perhaps they fixed it then.  I remember being it the reason I ported GNU
> sed to sabotage...
>
> GNU sed is one of the most unproblematic GNU tools, btw

Of the GNU tools, gawk, gmake and gsed all build without much problem under
the Plan9 APE (ANSI POSIX ENVIRONMENT) - so they are pretty portable and
independent of the whole GNU echosystem.

[-- Attachment #2: Type: text/html, Size: 557 bytes --]

Re: [Vision for new platform] syslog, sed, cron

[orc]

On Thu, 23 Aug 2012 11:15:07 +0200
Jens Staal <staal1978@gmail.com> wrote:

> Den 22 aug 2012 22:09 skrev "Christian Neukirchen"
> <chneukirchen@gmail.com>:
> 
> >
> > Perhaps they fixed it then.  I remember being it the reason I
> > ported GNU sed to sabotage...
> >
> > GNU sed is one of the most unproblematic GNU tools, btw
> 
> Of the GNU tools, gawk, gmake and gsed all build without much problem
> under the Plan9 APE (ANSI POSIX ENVIRONMENT) - so they are pretty
> portable and independent of the whole GNU echosystem.

The talk is about how their features can be emulated, not how gnu code
is portable?
(Why anyone who uses Plan9 needs gnu tools?)

Re: [Vision for new platform] syslog, sed, cron

[Rich Felker]

On Thu, Aug 23, 2012 at 05:33:34PM +0800, orc wrote:
> On Thu, 23 Aug 2012 11:15:07 +0200
> Jens Staal <staal1978@gmail.com> wrote:
> 
> > Den 22 aug 2012 22:09 skrev "Christian Neukirchen"
> > <chneukirchen@gmail.com>:
> > 
> > >
> > > Perhaps they fixed it then.  I remember being it the reason I
> > > ported GNU sed to sabotage...
> > >
> > > GNU sed is one of the most unproblematic GNU tools, btw
> > 
> > Of the GNU tools, gawk, gmake and gsed all build without much problem
> > under the Plan9 APE (ANSI POSIX ENVIRONMENT) - so they are pretty
> > portable and independent of the whole GNU echosystem.
> 
> The talk is about how their features can be emulated, not how gnu code
> is portable?
> (Why anyone who uses Plan9 needs gnu tools?)

If the discussion is on how to avoid using a portable, FOSS program
for ideological reasons, that seems, well, valid, but pretty different
from the goal of developing the new platform.

If it's not for ideological reasons but because the GNU programs
introduce unwanted or non-conformant behavior in some ways but are
needed for application compatibility in other ways, then it's a lot
more relevant..

Rich

Re: [Vision for new platform] syslog, sed, cron

[Kurt H Maier]

On Thu, Aug 23, 2012 at 05:33:34PM +0800, orc wrote:
> (Why anyone who uses Plan9 needs gnu tools?)

To build python, of course.  How else can we access mercurial?

Rich,  how many licenses will this platform require?

Re: [Vision for new platform] syslog, sed, cron

[Rich Felker]

On Thu, Aug 23, 2012 at 08:23:02AM -0400, Kurt H Maier wrote:
> On Thu, Aug 23, 2012 at 05:33:34PM +0800, orc wrote:
> > (Why anyone who uses Plan9 needs gnu tools?)
> 
> To build python, of course.  How else can we access mercurial?
> 
> Rich,  how many licenses will this platform require?

Good question. Again, the intent is not to impose policy
unnecessarily, so in cases where there are two or more implementations
of a standard utility that meet the common extension needs of
applications, any of them can be used at the system integrator or
distributor's discression. This mostly covers the standard utilities
but maybe not all the higher-level stuff where there may only be one
implementation of the software needed.

I suspect some users may prefer to have a fully BSD/MIT/permissive
userspace, for either ideological or misguided corporate policy
reasons. While misguided, at least the latter have something to do
with deployment and adoption, so it would be nice to have a complete
functional system with all the components we mentioned originally
under a permissive license, but I don't think having some significant
components under a different free license, at least initially, is a
show-stopping problem. Platform should excel on technical merits like
size, power consumption, etc. and usability merits like interface, not
on holier-than-thou attitudes about licenses...

Rich