Best place to discuss other lightweight libraries?

Best place to discuss other lightweight libraries?

[LM]

Musl does a great job of replacing glibc.  It's lightweight,
well-designed and written with friendly licensing.  However, I'm
noticing there are a lot of other standard libraries and tools on a
typical system and often many of them are bloated or not so well
designed.  I've been looking for a good forum to discuss using
alternative libraries and tools.  I've checked out projects like
Beyond Linux from Scratch and suckless.org/Plan 9 and several other
options.  I've even checked out some embedded systems mailing
lists/forums.  Have yet to find a good forum to discuss topics like
these with interested developers.  Does anyone know of a mailing list,
forum, etc. where one could discuss such topics.  If so, please let me
know.  If not, and if anyone on the list is interested in discussing
this sort of thing further, please e-mail me.  At the moment, I'm
investigating finding a less bloated alternative to libintl.  Did
notice gettext-tiny as part of Sabotage.  I've also run across 3 other
options and am looking at pros and cons of the implementations.

Thanks.

Sincerely,
Laura

Re: Best place to discuss other lightweight libraries?

[Rob Landley]

On 04/21/2013 11:30:54 AM, LM wrote:
> Musl does a great job of replacing glibc.  It's lightweight,
> well-designed and written with friendly licensing.  However, I'm
> noticing there are a lot of other standard libraries and tools on a
> typical system and often many of them are bloated or not so well
> designed.  I've been looking for a good forum to discuss using
> alternative libraries and tools.  I've checked out projects like
> Beyond Linux from Scratch and suckless.org/Plan 9 and several other
> options.  I've even checked out some embedded systems mailing
> lists/forums.  Have yet to find a good forum to discuss topics like
> these with interested developers.  Does anyone know of a mailing list,
> forum, etc. where one could discuss such topics.  If so, please let me
> know.  If not, and if anyone on the list is interested in discussing
> this sort of thing further, please e-mail me.  At the moment, I'm
> investigating finding a less bloated alternative to libintl.  Did
> notice gettext-tiny as part of Sabotage.  I've also run across 3 other
> options and am looking at pros and cons of the implementations.

If you find one, I'm interested in hearing about it.

In theory the linux-embedded list might be a good place:

   http://vger.kernel.org/vger-lists.html#linux-embedded

In practice, not much goes on there and the last post to that list was  
January. But if people did start posting there, I could join in. :)

Rob

Re: Best place to discuss other lightweight libraries?

[Rob Landley]

On 04/21/2013 11:30:54 AM, LM wrote:
> Musl does a great job of replacing glibc.  It's lightweight,
> well-designed and written with friendly licensing.  However, I'm
> noticing there are a lot of other standard libraries and tools on a
> typical system and often many of them are bloated or not so well
> designed.  I've been looking for a good forum to discuss using
> alternative libraries and tools.

Another place to ask would be:

   http://lists.celinuxforum.org/mailman/listinfo/celinux-dev

Which is the mailing list for the elinux.org wiki. It used to be Tim  
Bird's Consumer Electronic Linux Forum project, but unfortunately the  
Linux Foundation gobbled up CELF and turned it into The Linux  
Foundation Embedded Linux Conference By The Linux Foundation, and again  
discussion on the list tailed off.

It does have the advantage that it's got developers for actual  
electronics manufacturers (mostly japanese) subscribed to it...

Once upon a time busybox.net was collecting some of these:

   http://busybox.net/tinyutils.html

But that largely seems to have rolled to a stop since I handed it off...

The group of guys on #edev on freenode are quite knowledgeable about  
hardware stuff...

Rob

Re: Best place to discuss other lightweight libraries?

[LM]

[-- Attachment #1: Type: text/plain, Size: 1158 bytes --]

On Sun, Apr 21, 2013 at 4:24 PM, Rob Landley <rob@landley.net> wrote:

> Once upon a time busybox.net was collecting some of these:
>
>   http://busybox.net/tinyutils.html
>

Thanks for mentioning this.  I've seen the page before, but this was an
opportune time to go back and revisit it.  One thing on my todo list is to
find a non-interactive way to build Perl and microperl (as mentioned on the
page) appears to be just the solution I was looking for.

Would be curious if anyone's tried using the microperl makefile with musl.
I ran a search to see if I could dig up any more information on microperl.
I did see mention that the Perl developers weren't sure if they'd continue
to support it in the future.  Also found a couple of build scripts.  Both
install microperl executable and then create a link to perl using it.  Both
install various .pm files, but they each installed different sets of
files.  If anyone comes across documentation on installing microperl or has
any recommendations on which files are most useful to install and where to
put them, would be very interested in more details.

Thanks.

Sincerely,
Laura
http://www.distasis.com/cpp

[-- Attachment #2: Type: text/html, Size: 1609 bytes --]

Re: Best place to discuss other lightweight libraries?

[Rob Landley]

On 04/24/2013 06:39:56 AM, LM wrote:
> On Sun, Apr 21, 2013 at 4:24 PM, Rob Landley <rob@landley.net> wrote:
> 
> > Once upon a time busybox.net was collecting some of these:
> >
> >   http://busybox.net/tinyutils.html
> >
> 
> Thanks for mentioning this.  I've seen the page before, but this was  
> an
> opportune time to go back and revisit it.  One thing on my todo list  
> is to
> find a non-interactive way to build Perl and microperl (as mentioned  
> on the
> page) appears to be just the solution I was looking for.

The automated Linux From Scratch 6.8 build I did in Aboriginal Linux  
builds perl without requiring user interaction.

wget http://landley.net/aboriginal/bin/system-image-mips.tar.bz2 (or  
whichever)
tar xvjf system-image-mips.tar.bz2
cd system-image-mips
wget  
http://landley.net/aboriginal/control-images/downloads/binaries/lfs-bootstrap.hdc
./native-build.sh lfs-bootstrap.hdc

Does require qemu to be installed on the host...

(I don't _think_ that lfs-bootstrap binary is too stale. The  
system-image-mips is from the start of this month, when I finally got  
the darn arm bugs ironed out of the 3.8 kernel. I've been meaning to  
update to LFS 7.3 before cutting a new release, that one's 6.8. Plus a  
new dropbear binary came out for the static-tools.hdc build control  
image...)

However, I note all that currently builds against uClibc, not musl.  
(Day job eats time...)

> Would be curious if anyone's tried using the microperl makefile with  
> musl.

Microperl is part of the normal perl build. The perl build system is  
implemented _in_ perl (configure and make both), so first they build  
microperl to run the rest of it.

It's not really an intentionally shipped product, just a side effect of  
perl's head being up its own ass.

> I ran a search to see if I could dig up any more information on  
> microperl.
> I did see mention that the Perl developers weren't sure if they'd  
> continue
> to support it in the future.  Also found a couple of build scripts.   
> Both
> install microperl executable and then create a link to perl using  
> it.  Both
> install various .pm files, but they each installed different sets of
> files.  If anyone comes across documentation on installing microperl  
> or has
> any recommendations on which files are most useful to install and  
> where to
> put them, would be very interested in more details.

I'm under the vague impression that microperl is just a perl  
interpreter with several of the default libraries bundled into it, so  
it doesn't need to find anything in external search paths to run the  
perl configure and build.

I remember that people using it outside the perl build originally came  
as a surprise to the perl developers, but that was something like a  
decade ago and I expect they're used to it by now. Doesn't mean they  
put effort into supporting it...

Rob

Re: Best place to discuss other lightweight libraries?

[Isaac Dunham]

On Sun, 21 Apr 2013 12:30:54 -0400
LM <lmemsm@gmail.com> wrote:

> Musl does a great job of replacing glibc.  It's lightweight,
> well-designed and written with friendly licensing.  However, I'm
> noticing there are a lot of other standard libraries and tools on a
> typical system and often many of them are bloated or not so well
> designed.  I've been looking for a good forum to discuss using
> alternative libraries and tools.  I've checked out projects like
> Beyond Linux from Scratch and suckless.org/Plan 9 and several other
> options.  I've even checked out some embedded systems mailing
> lists/forums.  Have yet to find a good forum to discuss topics like
> these with interested developers.  Does anyone know of a mailing list,
> forum, etc. where one could discuss such topics.  If so, please let me
> know.  If not, and if anyone on the list is interested in discussing
> this sort of thing further, please e-mail me. 

FWIW, that's occasionally discussed on the Puppy Linux forums (programming subforum, some threads within Puppy Projects, and some discussion elsewhere...)
Some puplets are large, some are small, and most of the developers are after a way to build a small, fast, easy-to-use system (I know of several projects they have that target systems with less than 64 megs of RAM, and they're not tossing out hotplug)

This reminds me of this comment[1] from Flash, the moderator at the Puppy Linux forums:
You guys feel free to invite the developers to join our forum. 
Explain to them how useful Puppy could be to a developer.

I begged off at the time, but should probably mention it...
Although I use Puppy rather infrequently now (read: haven't booted that in several months, maybe a year), the Puppy forum is a better community than most I've run across.

[1] http://www.murga-linux.com/puppy/viewtopic.php?p=627636#627636

HTH,
Isaac Dunham

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Sun, Apr 21, 2013 at 12:30:54PM -0400, LM wrote:
> Musl does a great job of replacing glibc.  It's lightweight,
> well-designed and written with friendly licensing.  However, I'm
> noticing there are a lot of other standard libraries and tools on a
> typical system and often many of them are bloated or not so well
> designed.  I've been looking for a good forum to discuss using
> alternative libraries and tools.  I've checked out projects like

I have no objection to discussion on this list, at least for the time
being. If it becomes too much clutter, I might request later that we
move such discussions to a separate list, but for now, go ahead. We've
already had a few past discussions along these lines.

For what it's worth, I think poor design is really the core of the
problem. Bloat often stems from the effort needed to work around the
bad design. The major design issues I've seen in widespread libraries
are:

- Tacking on error handling as an afterthought or not at all. This
  leads to ugly things like the caller having to setup a jmp_buf for
  the library to bail out on error (likely leaking lots of memory in
  the process) or just aborting the program on error. Sometimes the
  hacks like longjmp are added in ways that they're not safe with
  multiple users of the library or with threads.

- Multiple "portable runtime" layers to do things the standard library
  already does, usually in less-portable or buggy ways.

- Making a big deal out of string handling. Ugly OO-in-C string types
  with dynamic allocation and slow concatenation operations all over
  the place. Basically, anything that's generating strings any way
  other than snprintf/asprintf, unless it's performance-critical, is
  broken.

- Gratuitous OO design. Good OO in C is having the complex or
  long-lived objects your library deals with well-encapsulated by
  library functions, and fully self-contained, without global state.
  Bad OO in C is reimplementing the C++ STL in C, with things like
  container objects, iterator objects, string objects, etc., as well
  as trying to do any global-level inheritance. As a clarification,
  things like implementing multiple video decoders that inherit an
  interface from a base "class" is reasonable; making every single
  object in your library derive from something like "GObject" is not.

- Gratuitous ugly typedefs, especially as part of the external API,
  and especially when they duplicate standard types (INT, gint, etc.).

- Interfaces that encourage or require UB to use them. One example
  that comes to mind is functions with signatures like posix_memalign
  which take a void** argument to store the result of an allocation.
  This encourages things like (void **)&ptr where ptr has type int*,
  for example.

- Thread allergies, i.e. horribly over-complicating program logic to
  avoid threads. The best examples I can think of are the added logic
  needed to generalize a program that's reading from ordinary file
  descriptors (e.g. connection sockets) in an event loop to support
  SSL sockets or zlib-compressed streams. (Note: there are ways to
  address this kind of problem more cleanly without threads too, but
  nobody does it. I can elaborate if anybody's interested.)

- DBus.

- Use of global state. Even seemingly-harmless things like a global
  registered log function are harmful, because two different libraries
  (or the main program and a library) might be trying to use the
  library with the global log destination, and clobbering each other's
  choices.

- Designs based on shared libraries, especially lots of them. This
  creates bloat and often interferes with the ability to use static
  linking.

- Excessive dependence on external files and filesystem layout. This
  conflicts with easily migrating the binaries to other machines.

- Dependency on any library with the above problems. :-)

Rich

Re: Best place to discuss other lightweight libraries?

[Luca Barbato]

On 04/22/2013 04:53 PM, Rich Felker wrote:
> - Thread allergies, i.e. horribly over-complicating program logic to
>   avoid threads. The best examples I can think of are the added logic
>   needed to generalize a program that's reading from ordinary file
>   descriptors (e.g. connection sockets) in an event loop to support
>   SSL sockets or zlib-compressed streams. (Note: there are ways to
>   address this kind of problem more cleanly without threads too, but
>   nobody does it. I can elaborate if anybody's interested.)

I'm interested to read about it.

> - DBus.

Sadly nobody is pushing for a better local socket multicast abstraction
to send notifications back and forth in an efficient fashion.

I'm hoping for nanomsg once it is complete or Binder once it is
correctly documented ^^; (and thus implemented in more than few forks of
linux and maybe haiku)

> - Use of global state. Even seemingly-harmless things like a global
>   registered log function are harmful, because two different libraries
>   (or the main program and a library) might be trying to use the
>   library with the global log destination, and clobbering each other's
>   choices.

For this there aren't solution that won't cause different problems I'm
afraid.

> - Designs based on shared libraries, especially lots of them. This
>   creates bloat and often interferes with the ability to use static
>   linking.

Special mention to those that want to do clever stuff on the init
section (e.g. change a program global state from there)


> - Dependency on any library with the above problems. :-)

And that kills everybody using glib? *runs and hides*

lu

Re: Best place to discuss other lightweight libraries?

[LM]

[-- Attachment #1: Type: text/plain, Size: 943 bytes --]

On Mon, Apr 22, 2013 at 11:21 AM, Luca Barbato <lu_zero@gentoo.org> wrote:

> > - Dependency on any library with the above problems. :-)
>
> And that kills everybody using glib? *runs and hides*
>

I guess that's one of the reasons I'm trying to eliminate glib from my
system and thus, at the moment, trying to find a gettext alternative that
doesn't need it.

Was reading about the Citrus Project ( http://citrus.bsdclub.org/ )
creating a BSD licensed gettext.  Having trouble digging up latest source
code for it though.  I also ran across Apache's version of iconv (
http://apr.apache.org/download.cgi ) but didn't see anything for gettext
mentioned at the site.

As time allows, I'm reworking sdcv which uses glib so that it doesn't need
it.  Then, all I need to do is find some useable replacements for SciTE and
Sylpheed and I think I will have eliminated any major needs for glib in the
Open Source applications and libraries I prefer.

[-- Attachment #2: Type: text/html, Size: 1337 bytes --]

Re: Best place to discuss other lightweight libraries?

[Daniel Cegiełka]

2013/4/22 LM <lmemsm@gmail.com>:
> On Mon, Apr 22, 2013 at 11:21 AM, Luca Barbato <lu_zero@gentoo.org> wrote:
>>
>> > - Dependency on any library with the above problems. :-)
>>
>> And that kills everybody using glib? *runs and hides*
>
>
> I guess that's one of the reasons I'm trying to eliminate glib from my
> system and thus, at the moment, trying to find a gettext alternative that
> doesn't need it.

https://github.com/rofl0r/gettext-tiny

Daniel

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Mon, Apr 22, 2013 at 06:47:02PM +0200, Daniel Cegiełka wrote:
> 2013/4/22 LM <lmemsm@gmail.com>:
> > On Mon, Apr 22, 2013 at 11:21 AM, Luca Barbato <lu_zero@gentoo.org> wrote:
> >>
> >> > - Dependency on any library with the above problems. :-)
> >>
> >> And that kills everybody using glib? *runs and hides*
> >
> > I guess that's one of the reasons I'm trying to eliminate glib from my
> > system and thus, at the moment, trying to find a gettext alternative that
> > doesn't need it.
> 
> https://github.com/rofl0r/gettext-tiny

I think the goal is to have gettext functionality, not just stubs.
Please correct me if I'm mistaken.

With that said, an old version of GNU gettext should work fine. I
wasn't even aware that the current version depends on glib; it
certainly didn't in the past.

I have certainly considered at times including a working gettext
implementation in musl or writing one for external use, but haven't
gotten around to it. There's no reason for it to be anything but tiny.
It doesn't do anything but search for and return strings out of
mmapped files.

Rich

Re: Best place to discuss other lightweight libraries?

[LM]

[-- Attachment #1: Type: text/plain, Size: 4715 bytes --]

Rich Felker wrote:
>On Mon, Apr 22, 2013 at 06:47:02PM +0200, Daniel Cegiełka wrote:
>>
>> https://github.com/rofl0r/gettext-tiny
>I think the goal is to have gettext functionality, not just stubs.
>Please correct me if I'm mistaken.
>
>With that said, an old version of GNU gettext should work fine. I
>wasn't even aware that the current version depends on glib; it
>certainly didn't in the past.

Correct, I was looking for a functional drop-in replacement that was better
designed or at least needed less dependencies.  Gettext was looking for
libxml2, libcroco and glib or would happily use its own versions if not
supplied.  It also has a circular reference with libiconv on some
platforms.  Was hoping for something more lightweight or at least with less
dependencies.

Wasn't aware that older versions didn't have all those dependencies.  That
definitely gives me one more alternative.  I did find an old version of bsd
gettext (
http://www.postgresql.org/message-id/Pine.LNX.4.30.0105222006170.757-100000@peter.localdomain)
that appears to be able to replace libintl and gettext.  It doesn't
supply replacements for msgfmt, msgmerge, xgettext.  Was checking if
gettext-tiny might replace them, but doesn't seem to supply the
functionality used by the Open Source program I was trying to compile.
Doesn't even get past the configure tests.  Might look into the possibility
of combining the older bsd gettext with any current modifications that
might look useful from BSD Citrus project.  Not sure if what I currently
have works properly for every language, but it at least appears to be
working for what I tested and it's a start.

Luca Barbato wrote:
> https://github.com/pkgconf/pkgconf might be handy btw.

Was going to mention that yesterday as well.  pkgconf is a great drop-in
replacement for pkgconfig and does not require glib or any circular library
references.  The developers of pkgconf were also nice about adding support
and/or receiving patches.  I absolutely cannot say the same thing about the
glib developers.

Rich Felker wrote:
>If you want the data structures, I think that means you should be using
C++, not C.

This may be very much a matter of taste, because you can write data
structures in any language, but that certainly sums up my preferences.  I
find C++ a very useful language for expressing data structures.

Rob Landley wrote:
>Many moons ago I started a thread on here (or was it on freenode?) asking
about lightweight alternatives to stuff and the need for a wiki page
tracking them.

I try to document them as I have time at my own web site.  I also add links
to the MinGW and OpenWatcom wikis if I find libraries or tools that might
be useful to developers.  I know there are some die-hard Linux advocates on
this list.  My own bias is toward cross-platform portability.  I started
out and still am a cross-platform programmer.  I prefer to be able to write
programs in such a way that they're easy to port to any machine.  I don't
like to be locked into only being able to use one operating system.
Sometimes one doesn't have a choice at work which operating system one is
stuck with and it's nice to be able to have Open Source tools and utilities
to work on any system.

Here's what I've dug up for MinGW as far as various tools, utilities and
libraries:
http://www.mingw.org/wiki/Community_Supplied_Links
The links are definitely slanted toward working with MinGW, but as I
mentioned, my preference is for cross-platform, so many of the libraries
and some of the utilities links work on a variety of operating systems.  A
few of them came from Linux/Unix environments and I had to send in patches
to port before they'd even work with MinGW.

Rob Landley wrote:
>I know we discussed more stuff (rxvt, xcfe and lxde...)

I'll throw in a mention of razor-qt and Equinox Desktop Environment as
lighter-weight desktop environment alternatives.  Personally, I've never
found xfce efficient on my older machine.  I currently prefer a lightweight
window manager and a few utilities over a desktop.  I also like
rxvt-unicode.  The ability to run multiple terminal windows with a daemon
in order to save memory is really nice.  The other terminals with this
feature (like LXTerminal, lilyterm, evilvte and Sakura) all seem to require
VTE and don't seem as efficient in some ways.

Connochaetos has some helpful memory usage information for comparing
various desktop and X applications.  Gives some statistics of rxvt-unicode
versus other terminals too in case anyone's curious.
http://www.connochaetos.org/wiki/devel:x-apps

Sincerely,
Laura
http://www.distasis.com/cpp/osrclist.htm

[-- Attachment #2: Type: text/html, Size: 5331 bytes --]

Re: Best place to discuss other lightweight libraries?

[John Spencer]

On 04/23/2013 02:50 PM, LM wrote:
> Rich Felker wrote:
>> On Mon, Apr 22, 2013 at 06:47:02PM +0200, Daniel Cegiełka wrote:
>>>
>>> https://github.com/rofl0r/gettext-tiny
>> I think the goal is to have gettext functionality, not just stubs.
>> Please correct me if I'm mistaken.
>>
>> With that said, an old version of GNU gettext should work fine. I
>> wasn't even aware that the current version depends on glib; it
>> certainly didn't in the past.
>
> Correct, I was looking for a functional drop-in replacement that was better
> designed or at least needed less dependencies.
[...]
> I did find an old version of bsd
> gettext (
> http://www.postgresql.org/message-id/Pine.LNX.4.30.0105222006170.757-100000@peter.localdomain)
> that appears to be able to replace libintl and gettext.  It doesn't
> supply replacements for msgfmt, msgmerge, xgettext.  Was checking if
> gettext-tiny might replace them, but doesn't seem to supply the

gettext-tiny is currently working as a stub that bounces back the 
original english message.
however msgfmt and msgmerge use a full blown .po parser that can easily 
be adapted to do the translation thing.
you are welcome to do the work (as long as you dont use C++)!


> functionality used by the Open Source program I was trying to compile.
> Doesn't even get past the configure tests.

which program was that ?
the main idea behind gettext-tiny is that it should make all configure 
scripts happy. so this is a bug, please report details here or open an 
issue on the github repo.

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Tue, Apr 23, 2013 at 04:40:48PM +0200, John Spencer wrote:
> >functionality used by the Open Source program I was trying to compile.
> >Doesn't even get past the configure tests.
> 
> which program was that ?
> the main idea behind gettext-tiny is that it should make all
> configure scripts happy. so this is a bug, please report details
> here or open an issue on the github repo.

Some programs need the gettext tools for .po/.mo files. I've found
that making an empty shell script serves as a suitable replacement for
them if you don't need message translation.

Rich

Re: Best place to discuss other lightweight libraries?

[Luca Barbato]

On 04/22/2013 06:40 PM, LM wrote:
> I guess that's one of the reasons I'm trying to eliminate glib from my
> system and thus, at the moment, trying to find a gettext alternative that
> doesn't need it.

Well eina from the enlightenment people isn't half bad if you need some
data structures but I doubt people would replace their working code
leveraging glib to use something else.

> As time allows, I'm reworking sdcv which uses glib so that it doesn't need
> it.  Then, all I need to do is find some useable replacements for SciTE and
> Sylpheed and I think I will have eliminated any major needs for glib in the
> Open Source applications and libraries I prefer.

https://github.com/pkgconf/pkgconf might be handy btw.

Re: Best place to discuss other lightweight libraries?

[Rob Landley]

On 04/22/2013 11:40:37 AM, LM wrote:
> On Mon, Apr 22, 2013 at 11:21 AM, Luca Barbato <lu_zero@gentoo.org>  
> wrote:
> 
> > > - Dependency on any library with the above problems. :-)
> >
> > And that kills everybody using glib? *runs and hides*
> >
> 
> I guess that's one of the reasons I'm trying to eliminate glib from my
> system and thus, at the moment, trying to find a gettext alternative  
> that
> doesn't need it.
> 
> Was reading about the Citrus Project ( http://citrus.bsdclub.org/ )
> creating a BSD licensed gettext.  Having trouble digging up latest  
> source
> code for it though.  I also ran across Apache's version of iconv (
> http://apr.apache.org/download.cgi ) but didn't see anything for  
> gettext
> mentioned at the site.

I use http://penma.de/code/gettext-stub/ which doesn't fix the problem,  
but does avoid it for my use cases. An actual desktop should be  
internationalizable, though.

> As time allows, I'm reworking sdcv which uses glib so that it doesn't  
> need
> it.  Then, all I need to do is find some useable replacements for  
> SciTE and
> Sylpheed and I think I will have eliminated any major needs for glib  
> in the
> Open Source applications and libraries I prefer.

Many moons ago I started a thread on here (or was it on freenode?)  
asking about lightweight alternatives to stuff and the need for a wiki  
page tracking them.

I believe that at the time, the musl wiki was insufficiently  
combobulated, but it has since been fixed. The  
http://busybox.net/tinyutils.html page is old is stale and never really  
had good coverage, http://elinux.org/System_Size is sort of adjacent to  
the topic. If we do come up with anything, can we put it on a wiki page?

Rob

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Mon, Apr 22, 2013 at 06:24:06PM -0500, Rob Landley wrote:
> On 04/22/2013 11:40:37 AM, LM wrote:
> >On Mon, Apr 22, 2013 at 11:21 AM, Luca Barbato
> ><lu_zero@gentoo.org> wrote:
> >
> >> > - Dependency on any library with the above problems. :-)
> >>
> >> And that kills everybody using glib? *runs and hides*
> >>
> >
> >I guess that's one of the reasons I'm trying to eliminate glib from my
> >system and thus, at the moment, trying to find a gettext
> >alternative that
> >doesn't need it.
> >
> >Was reading about the Citrus Project ( http://citrus.bsdclub.org/ )
> >creating a BSD licensed gettext.  Having trouble digging up latest
> >source
> >code for it though.  I also ran across Apache's version of iconv (
> >http://apr.apache.org/download.cgi ) but didn't see anything for
> >gettext
> >mentioned at the site.
> 
> I use http://penma.de/code/gettext-stub/ which doesn't fix the
> problem, but does avoid it for my use cases. An actual desktop
> should be internationalizable, though.

Agreed.

> >As time allows, I'm reworking sdcv which uses glib so that it
> >doesn't need
> >it.  Then, all I need to do is find some useable replacements for
> >SciTE and
> >Sylpheed and I think I will have eliminated any major needs for
> >glib in the
> >Open Source applications and libraries I prefer.
> 
> Many moons ago I started a thread on here (or was it on freenode?)
> asking about lightweight alternatives to stuff and the need for a
> wiki page tracking them.
> 
> I believe that at the time, the musl wiki was insufficiently
> combobulated, but it has since been fixed. The
> http://busybox.net/tinyutils.html page is old is stale and never
> really had good coverage, http://elinux.org/System_Size is sort of
> adjacent to the topic. If we do come up with anything, can we put it
> on a wiki page?

That would be a perfectly acceptable topic for the wiki.

Rich

Re: Best place to discuss other lightweight libraries?

[Rob Landley]

On 04/22/2013 06:31:10 PM, Rich Felker wrote:
> On Mon, Apr 22, 2013 at 06:24:06PM -0500, Rob Landley wrote:
> > Many moons ago I started a thread on here (or was it on freenode?)
> > asking about lightweight alternatives to stuff and the need for a
> > wiki page tracking them.
> >
> > I believe that at the time, the musl wiki was insufficiently
> > combobulated, but it has since been fixed. The
> > http://busybox.net/tinyutils.html page is old is stale and never
> > really had good coverage, http://elinux.org/System_Size is sort of
> > adjacent to the topic. If we do come up with anything, can we put it
> > on a wiki page?
> 
> That would be a perfectly acceptable topic for the wiki.

Obviously I'd nominate toybox. (And busybox.) If only so you have a  
context for "ok, what do these NOT cover".

There's an existing "musl vs uClibc" page but we might want a sentence  
or two on "musl: obviously the best", "uClibc: tried really hard but  
buildroot squished it in 2005 and it never recovered". "dietlibc:  
widely mocked for not-a-but-thats-a-feature disease", "klibc: official  
libc of the postminimalist art movement".)

"There's always room for dropbear". And polarssl, and so on.

I know we discussed more stuff (rxvt, xcfe and lxde...)

Rob

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Mon, Apr 22, 2013 at 07:54:55PM -0500, Rob Landley wrote:
> On 04/22/2013 06:31:10 PM, Rich Felker wrote:
> >On Mon, Apr 22, 2013 at 06:24:06PM -0500, Rob Landley wrote:
> >> Many moons ago I started a thread on here (or was it on freenode?)
> >> asking about lightweight alternatives to stuff and the need for a
> >> wiki page tracking them.
> >>
> >> I believe that at the time, the musl wiki was insufficiently
> >> combobulated, but it has since been fixed. The
> >> http://busybox.net/tinyutils.html page is old is stale and never
> >> really had good coverage, http://elinux.org/System_Size is sort of
> >> adjacent to the topic. If we do come up with anything, can we put it
> >> on a wiki page?
> >
> >That would be a perfectly acceptable topic for the wiki.
> 
> Obviously I'd nominate toybox. (And busybox.) If only so you have a
> context for "ok, what do these NOT cover".

Of course. My proposed criteria would just be:

1. Must be free software.

2. Duplicates a significant portion of the usage cases of a program or
library that's widely perceived as bloated or otherwise problematic
for systems musl might be used on, in a way that fixes some or all of
these problems. Reasons other than bloat might be waking up every
second to eat battery (non-Busybox ntpd does this!), requiring dynamic
linking, etc.

3. Non-criterion: the software doesn't have to be perfect or lack any
bloat problems itself; it just has to be better than the mainstream
solution in at least one way that might be significant to our users.

4. Not by Lennart Poettering. :-)

Does this sound reasonable?

> There's an existing "musl vs uClibc" page but we might want a
> sentence or two on "musl: obviously the best", "uClibc: tried really
> hard but buildroot squished it in 2005 and it never recovered".
> "dietlibc: widely mocked for not-a-but-thats-a-feature disease",
> "klibc: official libc of the postminimalist art movement".)

If such changes are going to be made, I think they should be done by
somebody who's not going to word them in that way... :-)

> "There's always room for dropbear". And polarssl, and so on.

cyassl looked promising too. I would probably mention tomcrypt too
even though it's not sufficient to do SSL; it has the most slim,
clean, portable implementations of crypto algorithms I've seen.

> I know we discussed more stuff (rxvt, xcfe and lxde...)

These are a bit more borderline, but I wouldn't call them
unacceptable.

Rich

Re: Best place to discuss other lightweight libraries?

[Isaac Dunham]

On Mon, 22 Apr 2013 21:46:40 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> 
> > "There's always room for dropbear". And polarssl, and so on.
> 
> cyassl looked promising too. I would probably mention tomcrypt too
> even though it's not sufficient to do SSL; it has the most slim,
> clean, portable implementations of crypto algorithms I've seen.

wpa_supplicant can use tomcrypt (external or internal) as fallback if no other encryption method (ie, openssl/gnutls) is configured, so I'd say it merits a mention.

I wonder if some notes should be put somewhere to point out that a network mangler on top of wpa_supplicant is not needed (the learning curve for configuring it is pretty steep, due to the need to find and understand the docs, but wpa_supplicant + wpa_cli -a script + wpa_cli in command mode can handle most situations, including dhcp).
I mention this because it seems to be "accepted wisdom" (but false) that you need wpa_supplicant as a tool and a network manager to make it useable.  And most of the network managers I've encountered are bloat of the highest order:
NetworkManager, wicd, wifiradar...
But this might be better put somewhere else.
--
Isaac Dunham

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Mon, Apr 22, 2013 at 10:04:30PM -0700, Isaac Dunham wrote:
> On Mon, 22 Apr 2013 21:46:40 -0400
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > 
> > > "There's always room for dropbear". And polarssl, and so on.
> > 
> > cyassl looked promising too. I would probably mention tomcrypt too
> > even though it's not sufficient to do SSL; it has the most slim,
> > clean, portable implementations of crypto algorithms I've seen.
> 
> wpa_supplicant can use tomcrypt (external or internal) as fallback
> if no other encryption method (ie, openssl/gnutls) is configured, so
> I'd say it merits a mention.

In that case I don't even see why they bother including the code to
use openssl/gnutls...

> I wonder if some notes should be put somewhere to point out that a
> network mangler on top of wpa_supplicant is not needed (the learning
> curve for configuring it is pretty steep, due to the need to find
> and understand the docs, but wpa_supplicant + wpa_cli -a script +
> wpa_cli in command mode can handle most situations, including dhcp).
> I mention this because it seems to be "accepted wisdom" (but false)
> that you need wpa_supplicant as a tool and a network manager to make
> it useable. And most of the network managers I've encountered are
> bloat of the highest order: NetworkManager, wicd, wifiradar... But
> this might be better put somewhere else.

Well the accepted wisdom is "almost true": for practical use of mobile
wifi, you need not just wpa_supplicant but also some controlling
process that's capable of:

1. Choosing which network to connect to.
2. Managing keys.
3. Logic for what to do when signal is lost.
4. Automating nonsense click-through agreements on public wifi.
...

The existing solutions all manage the above very poorly. Respectively,
they have:

1. No way to manage network priority/preference order.
2. Annoying popups to ask for key rather than having it be part of the
configuration of the network, and storing the keys in obscure places.
3. Annoying network-hopping.
4. Minimal or no auto-click-through; even when it does work, you can
get burned if your web browser happens to attempt a load before it
succeeds. A correct one needs to encapsulate the connection somehow so
that no connection is exposed to the user at all until the
click-through succeeds.

Rich

Re: Best place to discuss other lightweight libraries?

[Luca Barbato]

On 04/23/2013 03:47 PM, Rich Felker wrote:
> Well the accepted wisdom is "almost true": for practical use of mobile
> wifi, you need not just wpa_supplicant but also some controlling
> process that's capable of:
> 
> 1. Choosing which network to connect to.
> 2. Managing keys.
> 3. Logic for what to do when signal is lost.
> 4. Automating nonsense click-through agreements on public wifi.
> ...
> 
> The existing solutions all manage the above very poorly. Respectively,
> they have:
> 
> 1. No way to manage network priority/preference order.
> 2. Annoying popups to ask for key rather than having it be part of the
> configuration of the network, and storing the keys in obscure places.
> 3. Annoying network-hopping.
> 4. Minimal or no auto-click-through; even when it does work, you can
> get burned if your web browser happens to attempt a load before it
> succeeds. A correct one needs to encapsulate the connection somehow so
> that no connection is exposed to the user at all until the
> click-through succeeds.

connman is the closest I found so far, it uses dbus and glib so isn't
exactly a fit in this list.

lu

Re: Best place to discuss other lightweight libraries?

[Kurt H Maier]

On Tue, Apr 23, 2013 at 09:47:24AM -0400, Rich Felker wrote:
> 
> 1. Choosing which network to connect to.
> 2. Managing keys.
> 3. Logic for what to do when signal is lost.
> 4. Automating nonsense click-through agreements on public wifi.
> ...

wpa_supplicant can do 1., unless you mean choosing between ethernet and
wifi.  but it supports priority weighting on access points..

2. is pretty trivial since you can easily wpa_cli >> /etc/wpa.conf or 
similar.  I've never been sure why typing 'dhclient eth0' is seen as
more onerous than running a polling daemon to save you the trouble.

Can you elucidate more on 3?  if the signal is lost, wpa_supplicant
rescans and connects to any configured network, or else sleeps and
rescans later.

4. will never be solved satisfactorally, since that garbage is not
predictable.  the database of tedious TOS crap will never stop
expanding.

> 1. No way to manage network priority/preference order.

wpa_supplicant has priority= to do this.

> 2. Annoying popups to ask for key rather than having it be part of the
> configuration of the network, and storing the keys in obscure places.

What is less obscure than the wpa_supplicant config file?

> 3. Annoying network-hopping.

this can also be fixed with priority= 

> 4. Minimal or no auto-click-through; even when it does work, you can
> get burned if your web browser happens to attempt a load before it
> succeeds. A correct one needs to encapsulate the connection somehow so
> that no connection is exposed to the user at all until the
> click-through succeeds.

There are lots of useful things that can be done with this concept of an
encapsulated connection.  I get burned by this on my work laptop, which
likes to spam VPN connection attempts back to corporate.

khm

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Tue, Apr 23, 2013 at 05:50:23PM -0400, Kurt H Maier wrote:
> On Tue, Apr 23, 2013 at 09:47:24AM -0400, Rich Felker wrote:
> > 
> > 1. Choosing which network to connect to.
> > 2. Managing keys.
> > 3. Logic for what to do when signal is lost.
> > 4. Automating nonsense click-through agreements on public wifi.
> > ...
> 
> wpa_supplicant can do 1., unless you mean choosing between ethernet and
> wifi.  but it supports priority weighting on access points..

OK.

> 2. is pretty trivial since you can easily wpa_cli >> /etc/wpa.conf or 
> similar.  I've never been sure why typing 'dhclient eth0' is seen as
> more onerous than running a polling daemon to save you the trouble.

Because you don't have a keyboard, you have a 3-4" touchscreen. And
you want it to keep working as you move from place to place without
any interaction.

> Can you elucidate more on 3?  if the signal is lost, wpa_supplicant
> rescans and connects to any configured network, or else sleeps and
> rescans later.

I'm not sure what the ideal behavior is, but I know all the existing
ones have bad behavior.

> 4. will never be solved satisfactorally, since that garbage is not
> predictable.  the database of tedious TOS crap will never stop
> expanding.

Agree, but it still needs to be solved, even if the solution requires
frequent updates to be fully effective. With decent heuristics though
I think it could be fully automated for most sites with just a few
exceptions for really weird ones..

> > 4. Minimal or no auto-click-through; even when it does work, you can
> > get burned if your web browser happens to attempt a load before it
> > succeeds. A correct one needs to encapsulate the connection somehow so
> > that no connection is exposed to the user at all until the
> > click-through succeeds.
> 
> There are lots of useful things that can be done with this concept of an
> encapsulated connection.  I get burned by this on my work laptop, which
> likes to spam VPN connection attempts back to corporate.

Agreed. I think really most users should _always_ be running in an
environment where only root sees the real network interfaces and
applications just see a virtual network routed through the real one.

Rich

Re: Best place to discuss other lightweight libraries?

[Kurt H Maier]

On Tue, Apr 23, 2013 at 10:37:39PM -0400, Rich Felker wrote:
> 
> Because you don't have a keyboard, you have a 3-4" touchscreen. And
> you want it to keep working as you move from place to place without
> any interaction.
> 

If you don't have a keyboard, you likely also don't have physical
ethernet via rj-45, and even if you do, there's no reason to spam
wakeups checking if it's plugged in.  touch-only devices have touch-only
interfaces, which can easily include a 'switch to wired' button instead
of a daemon eating your battery.

setting physical links aside, wpa_supplicant can manage roaming fine.  I
should clarify that I think wpa_supplicant could do with a massive
wrecking ball, but it has a lot of good functionality that obviates a
ton of overengineered networkmanager-type software.  The trick will be
removing the garbage or implementing the good stuff in a slightly less
horrendous fashion.

In fact, I'm firmly of the opinion that complete signal loss is the
*only* time a system should monkey with the network; one of my least
favorite things is my phone aggressively dropping 3G so it can switch
to wifi, dumping my ssh sessions and filesystem mounts in the process. 

> Agree, but it still needs to be solved, even if the solution requires
> frequent updates to be fully effective. With decent heuristics though
> I think it could be fully automated for most sites with just a few
> exceptions for really weird ones..

I think the ideal solution is for network administrators to stop
pretending hijacking sessions is acceptable, but until an automated
solution exists I enjoy all the hate they get from users.

> Agreed. I think really most users should _always_ be running in an
> environment where only root sees the real network interfaces and
> applications just see a virtual network routed through the real one.

This doesn't necessarily solve anything from the user's standpoint
unless he's trained to use the feature appropriately, but it would
enable system designers to get uncomfortably clever in ways that can
make system behavior damn hard to predict.  Having said that, there is
software available on plan 9 called aan - 'any available network' - that
works this way and can be very useful.

khm

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Wed, Apr 24, 2013 at 12:43:06AM -0400, Kurt H Maier wrote:
> In fact, I'm firmly of the opinion that complete signal loss is the
> *only* time a system should monkey with the network; one of my least
> favorite things is my phone aggressively dropping 3G so it can switch
> to wifi, dumping my ssh sessions and filesystem mounts in the process. 

Ideally it would keep using both as long as there were some
"important" connections persisting on the old one, and there would be
a socket option for applications using unimportant persistent
connections to flag them unimportant.

> > Agree, but it still needs to be solved, even if the solution requires
> > frequent updates to be fully effective. With decent heuristics though
> > I think it could be fully automated for most sites with just a few
> > exceptions for really weird ones..
> 
> I think the ideal solution is for network administrators to stop
> pretending hijacking sessions is acceptable, but until an automated
> solution exists I enjoy all the hate they get from users.

Maybe once everyone finishes switching to https...then the hijacking
will cease to work, and to give a reasonable user experience, they'll
have to drop hijacking.

> > Agreed. I think really most users should _always_ be running in an
> > environment where only root sees the real network interfaces and
> > applications just see a virtual network routed through the real one.
> 
> This doesn't necessarily solve anything from the user's standpoint
> unless he's trained to use the feature appropriately, but it would

The assumption is that the system software, possibly interacting with
the user if the user were allowed to change network settings, would
handle the status of the real connection, and expose it only though
the virtual interface through the user when it's actually working. For
semi-advanced users, this could allow transparent migration (even
keeping your ssh/chat/etc. sessions) if you integrate it with vpn.

Rich

Re: Best place to discuss other lightweight libraries?

[idunham]

On Tue, Apr 23, 2013 at 09:47:24AM -0400, Rich Felker wrote:
> On Mon, Apr 22, 2013 at 10:04:30PM -0700, Isaac Dunham wrote:
> > On Mon, 22 Apr 2013 21:46:40 -0400
> > Rich Felker <dalias@aerifal.cx> wrote:
> > 
> > > 
> > > > "There's always room for dropbear". And polarssl, and so on.
> > > 
> > > cyassl looked promising too. I would probably mention tomcrypt too
> > > even though it's not sufficient to do SSL; it has the most slim,
> > > clean, portable implementations of crypto algorithms I've seen.
> > 
> > wpa_supplicant can use tomcrypt (external or internal) as fallback
> > if no other encryption method (ie, openssl/gnutls) is configured, so
> > I'd say it merits a mention.
> 
> In that case I don't even see why they bother including the code to
> use openssl/gnutls...

There are one or two features that need to be disabled to use tomcrypt.
I wish I could remember what they were.  But upstream has provided many
options that only duplicate functionality with additional bloat.
(sockets and plain C, vs. DBUS + glib)

> > I wonder if some notes should be put somewhere to point out that a
> > network mangler on top of wpa_supplicant is not needed (the learning
> > curve for configuring it is pretty steep, due to the need to find
> > and understand the docs, but wpa_supplicant + wpa_cli -a script +
> > wpa_cli in command mode can handle most situations, including dhcp).
> > I mention this because it seems to be "accepted wisdom" (but false)
> > that you need wpa_supplicant as a tool and a network manager to make
> > it useable. And most of the network managers I've encountered are
> > bloat of the highest order: NetworkManager, wicd, wifiradar... But
> > this might be better put somewhere else.
> 
> Well the accepted wisdom is "almost true": for practical use of mobile
> wifi, you need not just wpa_supplicant but also some controlling
> process that's capable of:
> 
> 1. Choosing which network to connect to.
Oh, like wpa_cli select_network ?

> 2. Managing keys.

wpa_cli [ passphrase | otp | password | new_password | pin | wps_pbc ]
(though figuring it out may be difficult, even with the help messages)

> 3. Logic for what to do when signal is lost.
wpa_supplicant reassociates on non-user-specified disconnects, and
wpa_cli -a <script>
allows configuration of the commands to run on CONNECTED and
DISCONNECTED events.

> 4. Automating nonsense click-through agreements on public wifi.
> ...

Nothing for this, as far as I know.  (On the other hand, I tend to dislike
software that pretends that I agreed to something I never saw.
Weird, I know ;-). )

> The existing solutions all manage the above very poorly...
What's worse is how some of them handle changing networks.

wpa_supplicant comes with wpa_cli for a reason: you need to be able to
tell the existing process to change its configuration.
The WRONG way to do things is to create a new config file, start a new 
instance of wpa_supplicant using that config file, and leave the old
wpa_supplicant running.
(wicd, I hope you've figured that out by now.)

Of course, setting up wpa_supplicant so that wpa_cli works is not easy.
And while wpa_gui (the Qt interface that corresponds to wpa_cli) is
available, it needs as much preconfiguration as wpa_cli, and the UI
could use some improvement before it's easy to understand (I can follow
it readily, but that's after using wpa_cli without anything else for a year or
two).  
A tool capable of producing a functional wpa_supplicant.conf and
providing a gui corresponding to wpa_cli in functionality would handle
most scenarios.
Unfortunately, the existing tools tend not to do that; I should see how
ceni works sometime-it's the only one I know of and haven't tried yet.

--
Isaac Dunham

Re: Best place to discuss other lightweight libraries?

[Rob Landley]

On 04/23/2013 12:04:30 AM, Isaac Dunham wrote:

> I wonder if some notes should be put somewhere to point out that a
> network mangler on top of wpa_supplicant is not needed (the learning
> curve for configuring it is pretty steep, due to the need to find and
> understand the docs, but wpa_supplicant + wpa_cli -a script + wpa_cli
> in command mode can handle most situations, including dhcp).
> I mention this because it seems to be "accepted wisdom" (but false)
> that you need wpa_supplicant as a tool and a network manager to make
> it useable.  And most of the network managers I've encountered are
> bloat of the highest order:
> NetworkManager, wicd, wifiradar...
> But this might be better put somewhere else.

Split up the page when it gets big and unmanageable. Don't try to  
categorize information that's not there yet.

And example shell scripts can be nice. You if you don't use them, it  
documents "here's what you need to do".

Rob

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Mon, Apr 22, 2013 at 05:21:25PM +0200, Luca Barbato wrote:
> On 04/22/2013 04:53 PM, Rich Felker wrote:
> > - Thread allergies, i.e. horribly over-complicating program logic to
> >   avoid threads. The best examples I can think of are the added logic
> >   needed to generalize a program that's reading from ordinary file
> >   descriptors (e.g. connection sockets) in an event loop to support
> >   SSL sockets or zlib-compressed streams. (Note: there are ways to
> >   address this kind of problem more cleanly without threads too, but
> >   nobody does it. I can elaborate if anybody's interested.)
> 
> I'm interested to read about it.

Well the canonical multi-threaded way to deal with such cases would be
to replace your file descriptor with a pipe or socket file descriptor
connected to a thread that's doing all the translation work (SSL,
zlib, etc.). Then, your main event loop just sees a normal file
descriptor, so you don't have to invade your code that handles
streams/connections/whatever with a new abstraction framework around
file descriptors that also supports SSL or whatever. Not only are such
abstraction layers bloated; they also add a lot of places for bugs to
hide, and they're difficult to get right. For a great example from
just a few days ago, see Bitlbee bug #1046:

http://bugs.bitlbee.org/bitlbee/ticket/1046

But suppose you want to solve this problem without threads. Well, the
basic idea is to do the same thing, but put the translation layer in
your event loop rather than in a thread. That is, make the pipe or
socketpair like you would have done for the threaded solution, but
instead of creating a thread, register the file descriptors with an
event handler in your main event loop. For an example with SSL, when
data comes in on the SSL socket, first the SSL-processing even handler
gets fired, and writes the decrypted stream to a pipe or socketpair.
Then, the event handler for the other end of that pipe/socket runs and
handles it like an ordinary connection.

Obviously this has some degree of additional overhead compared to an
abstraction layer, so I wouldn't recommend doing it for applications
whose JOB is to handle connections or streaming data as efficiently as
possible (e.g. a webserver). But for something like an IRC client or
web browser, where socket performance is irrelevant compared to other
things, it's idiotic to design fancy abstraction layers for reading
and writing to connections when you could just do a design like what I
described above.

> > - DBus.
> 
> Sadly nobody is pushing for a better local socket multicast abstraction
> to send notifications back and forth in an efficient fashion.
> 
> I'm hoping for nanomsg once it is complete or Binder once it is
> correctly documented ^^; (and thus implemented in more than few forks of
> linux and maybe haiku)

Yes, Binder looks really promising, but I also think part of the
problem is that the need for this kind of interface is exaggerated...

> > - Use of global state. Even seemingly-harmless things like a global
> >   registered log function are harmful, because two different libraries
> >   (or the main program and a library) might be trying to use the
> >   library with the global log destination, and clobbering each other's
> >   choices.
> 
> For this there aren't solution that won't cause different problems I'm
> afraid.

Sure there are. I get the impression you can tell I was talking about
libav/ffmpeg's log interface. :-) The obvious solution is to bind log
contexts to the object you're acting on. See this nice talk:

http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/

If I remember right, part of the problem way back was that there were
deep function calls that had no context available to them, and that
didn't _need_ a context for anything but logging warnings or whatnot.
Really, the fact that they can fail and want to be able to report
failure means they _do_ need a context, but I can understand the
desire to cheat, especially if there's a performance cost to passing
the context all the way down. In that case, hidden non-global state,
namely thread-local storage, might be an appropriate solution. It's
still hidden state (and thus A Bad Thing), but at least it's no longer
global state.

> > - Designs based on shared libraries, especially lots of them. This
> >   creates bloat and often interferes with the ability to use static
> >   linking.
> 
> Special mention to those that want to do clever stuff on the init
> section (e.g. change a program global state from there)

Did whatever lib did that (OpenAL, was it?) ever fix their bugs?

> > - Dependency on any library with the above problems. :-)
> 
> And that kills everybody using glib? *runs and hides*

Yes, basically. Dependency on glib means your library will impose
bloat and it will preclude robustness.

Rich

Re: Best place to discuss other lightweight libraries?

[Luca Barbato]

On 04/22/2013 11:52 PM, Rich Felker wrote:
>> For this there aren't solution that won't cause different problems I'm
>> afraid.
> 
> Sure there are. I get the impression you can tell I was talking about
> libav/ffmpeg's log interface. :-) The obvious solution is to bind log
> contexts to the object you're acting on. See this nice talk:
> 
> http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/
> 
> If I remember right, part of the problem way back was that there were
> deep function calls that had no context available to them, and that
> didn't _need_ a context for anything but logging warnings or whatnot.

In the specific case yes. I tried to foster proper return error
propagation, so you get something more meaningful than EINVAL/-1 and
that is usually enough in those specific cases.

The general problem is that the library user wants to be the only one
having a say on what goes where so single point overrides are useful.

When you start using those libraries in situations in which you'd like
to have per-$situation logging then you start to scream.

(In the specific case above there is a quite borderline solution since
we can override the global logger and store per context loggers in
creative ways)

> Really, the fact that they can fail and want to be able to report
> failure means they _do_ need a context, but I can understand the
> desire to cheat, especially if there's a performance cost to passing
> the context all the way down. In that case, hidden non-global state,
> namely thread-local storage, might be an appropriate solution. It's
> still hidden state (and thus A Bad Thing), but at least it's no longer
> global state.

Exactly. (bonus points if you do that in a void returning function...)

>>> - Designs based on shared libraries, especially lots of them. This
>>>   creates bloat and often interferes with the ability to use static
>>>   linking.
>>
>> Special mention to those that want to do clever stuff on the init
>> section (e.g. change a program global state from there)
> 
> Did whatever lib did that (OpenAL, was it?) ever fix their bugs?

Given recent binutils would strip away symbols if used that way, making
linking fail, I hope so.

> Yes, basically. Dependency on glib means your library will impose
> bloat and it will preclude robustness.

Yet glib gives you oh-so-many-features (I fell for it once), sadly there
aren't many utility libs that provide sort of useful data structures,
eventloops, threadpools and portable string mangling (e.g. strl*
functions) in a single widespread package.

Some lean/cleaned up alternative are cropping but usually they aren't as
known and widespread.

lu

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Tue, Apr 23, 2013 at 12:42:01AM +0200, Luca Barbato wrote:
> On 04/22/2013 11:52 PM, Rich Felker wrote:
> >> For this there aren't solution that won't cause different problems I'm
> >> afraid.
> > 
> > Sure there are. I get the impression you can tell I was talking about
> > libav/ffmpeg's log interface. :-) The obvious solution is to bind log
> > contexts to the object you're acting on. See this nice talk:
> > 
> > http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/
> > 
> > If I remember right, part of the problem way back was that there were
> > deep function calls that had no context available to them, and that
> > didn't _need_ a context for anything but logging warnings or whatnot.
> 
> In the specific case yes. I tried to foster proper return error
> propagation, so you get something more meaningful than EINVAL/-1 and
> that is usually enough in those specific cases.
> 
> The general problem is that the library user wants to be the only one
> having a say on what goes where so single point overrides are useful.

The problem with your comment here is the phrase "library user". Who
is the library user? You may be thinking from a standpoint (in our
example) of MPlayer, but what if instead the application you were
looking at were a file manager that itself had no awareness of video
files, and only ended up processing them as part of a library pulled
in from a plugin for file previews? Obviously there is no way the app
can be aware of where the log messages should go since it's not aware
the library even exists. The user is the library that depends on the
library doing the logging, not the app, and it's very possible that
there is more than once such library. In which case, you have madness.

> When you start using those libraries in situations in which you'd like
> to have per-$situation logging then you start to scream.

Keep in mind it might not even be "per-situation logging". It might be
something like one plugin needing to send the message back up to the
application as a popup message to display, and another plugin just
wanting to render the message text as a file preview in place of an
image...

> > Yes, basically. Dependency on glib means your library will impose
> > bloat and it will preclude robustness.
> 
> Yet glib gives you oh-so-many-features (I fell for it once), sadly there
> aren't many utility libs that provide sort of useful data structures,

If you want the data structures, I think that means you should be
using C++, not C.

> eventloops,

If your program is event-loop-performance-critical (think httpd), you
probably need a custom one for your application.

If it's not, you could probably write a much simpler program using
threads. It might even perform better too.

> threadpools

Similar situation. Threadpools are an optimization that shouldn't be
applied prematurely, and if you do need them, you probably want a
custom solution.

> and portable string mangling (e.g. strl*
> functions) in a single widespread package.

strl* considered harmful, for 3 reasons:

1. Concatenation is a bad idiom. See
http://en.wikipedia.org/wiki/Schlemiel_the_Painter's_algorithm

2. It duplicates standard functionality in snprintf. Yes it's quicker
for some cases (pure copying), but in that case you probably should
just be using the original string in-place. On the other hand,
building the whole output in one step with snprintf makes it obvious
that your code is correct and avoids issue #1 above.

3. While returning the length that "would be needed" is useful if you
want to allocate and retry, it's harmful if your goal is to protect
against malicious inputs. For instance,

    strlcpy(buf, some_500_gb_string, 10)

has to process the whole input string even though it's going to throw
away most of it. BTW this issue applies to snprintf too, but at least
it has something useful to be computing. With strl*, the return value
is always something that would be trivial to obtain yourself if you
needed it.

I'm aware some people like strl* and we have them in musl because it's
better to provide them than to deal with people rolling their own and
doing it in wrong and insecure ways. But I still would recommend
against using them in new code.

Rich

Re: Best place to discuss other lightweight libraries?

[Luca Barbato]

On 04/23/2013 01:06 AM, Rich Felker wrote:
> On Tue, Apr 23, 2013 at 12:42:01AM +0200, Luca Barbato wrote:
>> On 04/22/2013 11:52 PM, Rich Felker wrote:
>>>> For this there aren't solution that won't cause different problems I'm
>>>> afraid.
>>>
>>> Sure there are. I get the impression you can tell I was talking about
>>> libav/ffmpeg's log interface. :-) The obvious solution is to bind log
>>> contexts to the object you're acting on. See this nice talk:
>>>
>>> http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/
>>>
>>> If I remember right, part of the problem way back was that there were
>>> deep function calls that had no context available to them, and that
>>> didn't _need_ a context for anything but logging warnings or whatnot.
>>
>> In the specific case yes. I tried to foster proper return error
>> propagation, so you get something more meaningful than EINVAL/-1 and
>> that is usually enough in those specific cases.
>>
>> The general problem is that the library user wants to be the only one
>> having a say on what goes where so single point overrides are useful.
> 
> The problem with your comment here is the phrase "library user". Who
> is the library user? You may be thinking from a standpoint (in our
> example) of MPlayer, but what if instead the application you were
> looking at were a file manager that itself had no awareness of video
> files, and only ended up processing them as part of a library pulled
> in from a plugin for file previews? Obviously there is no way the app
> can be aware of where the log messages should go since it's not aware
> the library even exists. The user is the library that depends on the
> library doing the logging, not the app, and it's very possible that
> there is more than once such library. In which case, you have madness.

Usually (at least that's what I do in those case) the global logger is
overridden to use the outer library logger then you -end-user- override
it as well and then everything goes where you want.

The other widespread solution is to eat stderr and if something appears
show to the user, crude but not so bad.

>> When you start using those libraries in situations in which you'd like
>> to have per-$situation logging then you start to scream.
> 
> Keep in mind it might not even be "per-situation logging". It might be
> something like one plugin needing to send the message back up to the
> application as a popup message to display, and another plugin just
> wanting to render the message text as a file preview in place of an
> image...

Yeah, logging messages properly is terrible.

>>> Yes, basically. Dependency on glib means your library will impose
>>> bloat and it will preclude robustness.
>>
>> Yet glib gives you oh-so-many-features (I fell for it once), sadly there
>> aren't many utility libs that provide sort of useful data structures,
> 
> If you want the data structures, I think that means you should be
> using C++, not C.

C++ stock data structures are too runtime-dependent, crafting your own
means getting the worst of both words if you aren't extremely careful =\

Hopefully the new crop of system languages would try to capitalize on
the experience...

>> eventloops,
> 
> If your program is event-loop-performance-critical (think httpd), you
> probably need a custom one for your application.
> 
> If it's not, you could probably write a much simpler program using
> threads. It might even perform better too.

event loops, coroutines and threads have their uses.

>> threadpools
> 
> Similar situation. Threadpools are an optimization that shouldn't be
> applied prematurely, and if you do need them, you probably want a
> custom solution.

There is always a tradeoff, reimplementing the wheel is ok if only
square ones are available.

>> and portable string mangling (e.g. strl*
>> functions) in a single widespread package.
> 
> strl* considered harmful, for 3 reasons:
> 
> 1. Concatenation is a bad idiom. See
> http://en.wikipedia.org/wiki/Schlemiel_the_Painter's_algorithm

scatter-gather lists aren't that widespread =)

> 2. It duplicates standard functionality in snprintf. Yes it's quicker
> for some cases (pure copying), but in that case you probably should
> just be using the original string in-place. On the other hand,
> building the whole output in one step with snprintf makes it obvious
> that your code is correct and avoids issue #1 above.

Using snprintf in those cases doesn't scream efficient if I recall
correctly.

> I'm aware some people like strl* and we have them in musl because it's
> better to provide them than to deal with people rolling their own and
> doing it in wrong and insecure ways. But I still would recommend
> against using them in new code.

That's more or less the annoying question: better to provide good
implementation of slightly dangerous to use code or let people bake
their own? (usually you have people using something not implemented
perfectly doesn't matter what, but again, tradeoffs...)

lu

Re: Best place to discuss other lightweight libraries?

[Rob Landley]

On 04/22/2013 07:26:21 PM, Luca Barbato wrote:
> On 04/23/2013 01:06 AM, Rich Felker wrote:
> > On Tue, Apr 23, 2013 at 12:42:01AM +0200, Luca Barbato wrote:
> >> On 04/22/2013 11:52 PM, Rich Felker wrote:
> >>>> For this there aren't solution that won't cause different  
> problems I'm
> >>>> afraid.
> >>>
> >>> Sure there are. I get the impression you can tell I was talking  
> about
> >>> libav/ffmpeg's log interface. :-) The obvious solution is to bind  
> log
> >>> contexts to the object you're acting on. See this nice talk:
> >>>
> >>>  
> http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/
> >>>
> >>> If I remember right, part of the problem way back was that there  
> were
> >>> deep function calls that had no context available to them, and  
> that
> >>> didn't _need_ a context for anything but logging warnings or  
> whatnot.
> >>
> >> In the specific case yes. I tried to foster proper return error
> >> propagation, so you get something more meaningful than EINVAL/-1  
> and
> >> that is usually enough in those specific cases.
> >>
> >> The general problem is that the library user wants to be the only  
> one
> >> having a say on what goes where so single point overrides are  
> useful.
> >
> > The problem with your comment here is the phrase "library user". Who
> > is the library user? You may be thinking from a standpoint (in our
> > example) of MPlayer, but what if instead the application you were
> > looking at were a file manager that itself had no awareness of video
> > files, and only ended up processing them as part of a library pulled
> > in from a plugin for file previews? Obviously there is no way the  
> app
> > can be aware of where the log messages should go since it's not  
> aware
> > the library even exists. The user is the library that depends on the
> > library doing the logging, not the app, and it's very possible that
> > there is more than once such library. In which case, you have  
> madness.
> 
> Usually (at least that's what I do in those case) the global logger is
> overridden to use the outer library logger then you -end-user-  
> override
> it as well and then everything goes where you want.
> 
> The other widespread solution is to eat stderr and if something  
> appears
> show to the user, crude but not so bad.
> 
> >> When you start using those libraries in situations in which you'd  
> like
> >> to have per-$situation logging then you start to scream.
> >
> > Keep in mind it might not even be "per-situation logging". It might  
> be
> > something like one plugin needing to send the message back up to the
> > application as a popup message to display, and another plugin just
> > wanting to render the message text as a file preview in place of an
> > image...
> 
> Yeah, logging messages properly is terrible.
> 
> >>> Yes, basically. Dependency on glib means your library will impose
> >>> bloat and it will preclude robustness.
> >>
> >> Yet glib gives you oh-so-many-features (I fell for it once), sadly  
> there
> >> aren't many utility libs that provide sort of useful data  
> structures,
> >
> > If you want the data structures, I think that means you should be
> > using C++, not C.
> 
> C++ stock data structures are too runtime-dependent, crafting your own
> means getting the worst of both words if you aren't extremely careful  
> =\
> 
> Hopefully the new crop of system languages would try to capitalize on
> the experience...

What new crop of system languages?

C is a portable assembly language with minimal abstraction between the  
programmer and what the hardware is actually doing. It uses static  
typing, static memory allocation, and if you really care you can  
explicitly specify integer sizes (uint16_t or LP64) and handle  
endianness and alignment and so on down to memory mapped bitmasks. It  
provides simple container types based on pointer math: arrays are  
simple pointer arithmetic, and structs concatenate a group of variables  
so each member name corresponds to a fixed offset and size (static,  
determined at compile time) where the value is to be found relative to  
the pointer to the start of the struct.

Scripting languages like python/lua/ruby have opaque abstractions where  
you honestly don't need to know how it's implemented. They replace  
poitners with references, and build garbage collection and dynamic  
typing on top of that. Their built-in container types are resizeable,  
including an array variant and a dictionary variant. The dictionaries  
aggregate via keyword/value association, so you can add and remove  
members on the fly.

In C, types are a property of pointers. In scripting languages, types  
are a property of objects, meaning _references_have_no_type_. You  
dereference to find out what type it is. So when you implement  
functions, you find out what type it is when you try to use it, but  
asking for a member and performing an operation on that member. If the  
member isn't there, or doesn't support that operation, it throws an  
exception. You can catch that exception and handle it however you like,  
up to and including adjusting the object to add the member in question  
so it _can_ succeed. But if you don't catch the exception locally, no  
problem: it's all garbage collected. References that fall out of scope  
are naturally freed by the system.

These are two fundamentally different ways of programming. scripting  
languages are dynamic, everything interesting determineda t runtime, to  
the point where they don't even have a compilation step. You set the  
executable bit on your source code. (Is there a bytecode compilation  
step at load time with an optimized interpreter doing batched code  
translation with buffering that Sun's marketing department called "just  
in time" or some such nonsense but which Apple's 68k emulator for the  
PowerPC was already doing in 1994? Maybe. Again: it doesn't matter, the  
abstractions are opaque, it all just works.)

So with C: pointers, everything statically compiled to machine  
language, no abstraction. With scripting langauges: references,  
interpreted at runtime, opaque abstraction and often multiple different  
but compatible implementations (python/jython).

Then you have C++, which can't decide which it is. C is a local peak in  
language design space. scripting languages are another. C++ is in the  
valley between them, neither fish nor fowl, with the worst  
characteristics of both. It's a static language like C, statically  
typed and based on pointers, with thick layers of _leaky_ abstractions.  
If anything goes wrong, you have to know all the implementation details  
of the standard template library in order to debug it. Your global  
constructors are called before main() and those have zeroed memory but  
when you new() an object it doesn't have zeroed memory and you must  
initialize every single member in the constructor and of coure you  
can't memset(this, 0, sizeof(this)) because there's magic data in the  
object for RTTI and virtual methods which you can't _see_ but which you  
can trivially damage if you don't know the magic invisible  
implementation details.

ALL of C++ is magic invisible implementation details. The only way to  
safely use the language is to know enough about it you could have  
written the compiler and all the libraries. Otherwise, it's going to  
break and you won't know why, although following magic "design  
patterns" from your local cargo cult leader may help shield you from  
the wrath of the compiler for another day, if you're lucky and turn  
widdershins twice every tuesday before noon but after having the  
_right_ cup of coffee while wearing lucky socks.

C++ saw scripting languages and tried to ape their features  
(Exceptions!) but doing dynamic typing at compile time is every bit as  
stupid as doing dynamic memory management at compile time, and their  
attempt (templates) is TURING COMPLETE AT COMPILE TIME meaning you can  
write 10 lines of C++ that will keep the compiler busy until your hard  
drive fills up, and detecting this is equivalent to solving the halting  
problem. Even when it does NOT do that, a couple lines of C++ template  
making your binary ten times larger is considered _normal_.

Note: Java is also in the no man's land between C and scripting  
languages, but it's in the foothills of scripting languages instead of  
the foothills of C: it did dynamic memory management but _kept_ static  
typing, then realized how dumb taht was and punched holes in its type  
system with "interfaces", and then made code generators to spit out  
reams of interface code and designed new tools (Eclipse!) to handle  
multi-million line code bases for 2 year old projects made by 3 people.  
Alas, when Y2K happened and all that Cobol needed to be rewritten Java  
was the hot new fad (Pogs! Beanie Babies! Bitcoin!) and looked good in  
comparison to cobol, so it's the new mainframe punchcard language. Oh  
well.

Steve Yegge eviscerated Java so I don't have to here:
http://steve-yegge.blogspot.com/2007/12/codes-worst-enemy.html

So back to the "new generation of system languages": C is a portable  
assembly language. It's a category killer in that niche, the best there  
is at what it does that's already killed off competitors like Pascal.  
The only real survivors are derivatives of C whose main selling point  
is that they CONTAIN THE WHOLE OF C, VERBATIM. (By that logic a mud pie  
is a good beverage, because each mud pie contains a glass of water.)

Scripting langauges (even ugly ones like Javascript, Perl, and PHP)  
rely on opaque abstractions independent of what the hardware is doing.  
Java is the new Cobol.

Which direction is your new system language going in?

> > strl* considered harmful, for 3 reasons:
...
> > I'm aware some people like strl* and we have them in musl because  
> it's
> > better to provide them than to deal with people rolling their own  
> and
> > doing it in wrong and insecure ways. But I still would recommend
> > against using them in new code.

Toybox has xstrncpy(): If the string doesn't fit in the buffer, kill  
the program with an error message.

Rob

Re: Best place to discuss other lightweight libraries?

[Strake]

On 22/04/2013, Rob Landley <rob@landley.net> wrote:
> [Java] did dynamic memory management but _kept_ static
> typing, then realized how dumb taht was

So they realized that it's a damn good idea, but somehow made a crap
language nevertheless.

Haskell, for example, does dynamic memory allocation with static
types, and quite often wins.

> So back to the "new generation of system languages": C is a portable
> assembly language. It's a category killer in that niche, the best there
> is at what it does that's already killed off competitors like Pascal.

So on that note, I deem Haskell would be a categorical category killer (^_^)

Cheers,
Strake

Re: Best place to discuss other lightweight libraries?

[Daniel Cegiełka]

2013/4/23 Strake <strake888@gmail.com>:
> So on that note, I deem Haskell would be a categorical category killer (^_^)
>
> Cheers,
> Strake


Haskell and musl - has anyone tried this combination? :) GHC is a
pretty big package.
Daniel

Re: Best place to discuss other lightweight libraries?

[Szabolcs Nagy]

* Daniel Cegie?ka <daniel.cegielka@gmail.com> [2013-04-23 21:24:57 +0200]:
> 2013/4/23 Strake <strake888@gmail.com>:
> > So on that note, I deem Haskell would be a categorical category killer (^_^)
> 
> Haskell and musl - has anyone tried this combination? :) GHC is a
> pretty big package.

as far as i know haskell has no well defined semantics
about its interaction with the c runtime eventhough it
has an ffi that can use c libraries directly
(eg how haskell managed threads interact with c threads)

(the same is true for most high level languages that also
try to do system level things)

it would be nice to look at these issues systematically

in c it's reasonably clear what the runtime does and what
it doesnt do and how the application can interact with
the system through the c runtime

in a high level language you interact with the system
through high level abstractions (implemented by the
language runtime that almost always goes through the
c runtime) *and* through the c runtime when external
c libraries are used which interact with the system
as well

if you only go through the high level language runtime
then in theory it can shield you from ugly details of
the system (in practice it's not so: you can call
exec, fork, etc from python/php/perl.. and they dont
fully protect the abstractions of the language:
destructors are not called when you exit with
os.exec('/bin/true'))

when the high level language runtime is mixed with the
c runtime through ffi that's a problem: you need
'c level' guarantees about the implementation of the
language runtime and those are the things language
designers want to hide from you

Re: Best place to discuss other lightweight libraries?

[Zvi Gilboa]

On 04/23/2013 05:33 PM, Szabolcs Nagy wrote:
> * Daniel Cegiełka <daniel.cegielka@gmail.com> [2013-04-23 21:24:57 +0200]:
>> 2013/4/23 Strake <strake888@gmail.com>:
>>> So on that note, I deem Haskell would be a categorical category killer (^_^)
>> Haskell and musl - has anyone tried this combination? :) GHC is a
>> pretty big package.

One aspect of language binding that often goes under the radar concerns 
the method to create public API header files. For my own projects, I use 
a simple PostgreSql database that stores the definitions of constants, 
structures, functions, and also the public API header-tree, and then a 
few shell/psql scripts to generate the entire set of public API headers. 
As a matter of convenience, I normally enter the information into plain 
text files, which are then processed from the command line and populate 
the tables of the above database. In my experience, that kind of 
framework not only makes it easy to add binding for new languages, but 
also simplifies book-keeping tasks such as the splitting or joining of 
libraries, API consistency checks, etc.

Re: Best place to discuss other lightweight libraries?

[Luca Barbato]

On 04/23/2013 04:14 AM, Rob Landley wrote:
> What new crop of system languages?

I could point Go and Rust as two that are trying too much for my taste.

There are few other that try to add just small syntactic features to C
(e.g. blocks/anonymous function) to make the language a bit more
expressive and yet compile to something abi/api compatible with C.

> Toybox has xstrncpy(): If the string doesn't fit in the buffer, kill the
> program with an error message.

That's fine for a program, a horrid sin for a library.

> Rob

Re: Best place to discuss other lightweight libraries?

[Daniel Cegiełka]

2013/4/23 Luca Barbato <lu_zero@gentoo.org>:
> On 04/23/2013 04:14 AM, Rob Landley wrote:
>> What new crop of system languages?
>
> I could point Go and Rust as two that are trying too much for my taste.

rust - ok, but written in c++; llvm as dependency.

go - nice, only ed as a dependency but really stupid build system:

https://code.google.com/p/go/source/browse#hg%2Fsrc%2Fcmd%2Fdist

https://code.google.com/p/go/source/browse/src/cmd/dist/build.c

btw. has anyone used go with musl?

Daniel

Re: Best place to discuss other lightweight libraries?

[Kurt H Maier]

On Wed, Apr 24, 2013 at 01:18:43PM +0200, Daniel Cegiełka wrote:
> 
> btw. has anyone used go with musl?
> 

Go ships its own libc, which I'm fairly certain it depends on.  It's
also not suitable as a system programming language and they dropped that
claim from their propaganda some time ago.

khm

Re: Best place to discuss other lightweight libraries?

[Daniel Cegiełka]

2013/4/24 Kurt H Maier <khm-lists@intma.in>:
> On Wed, Apr 24, 2013 at 01:18:43PM +0200, Daniel Cegiełka wrote:
>>
>> btw. has anyone used go with musl?
>>
>
> Go ships its own libc, which I'm fairly certain it depends on.

lib9, but are you sure about that?

# ldd /usr/bin/go
	linux-vdso.so.1 =>  (0x00007fff50fff000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff215e4b000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ff215abf000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff216068000)

Daniel

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Wed, Apr 24, 2013 at 02:32:21PM +0200, Daniel Cegiełka wrote:
> 2013/4/24 Kurt H Maier <khm-lists@intma.in>:
> > On Wed, Apr 24, 2013 at 01:18:43PM +0200, Daniel Cegiełka wrote:
> >>
> >> btw. has anyone used go with musl?
> >>
> >
> > Go ships its own libc, which I'm fairly certain it depends on.
> 
> lib9, but are you sure about that?
> 
> # ldd /usr/bin/go
> 	linux-vdso.so.1 =>  (0x00007fff50fff000)
> 	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff215e4b000)
> 	libc.so.6 => /lib64/libc.so.6 (0x00007ff215abf000)
> 	/lib64/ld-linux-x86-64.so.2 (0x00007ff216068000)

Is the go binary written in go? The point was that go links programs
against its own libc, not that the go compiler is linked against its
own libc.

Rich

Re: Best place to discuss other lightweight libraries?

[Daniel Cegiełka]

2013/4/24 Rich Felker <dalias@aerifal.cx>:
> On Wed, Apr 24, 2013 at 02:32:21PM +0200, Daniel Cegiełka wrote:
>> 2013/4/24 Kurt H Maier <khm-lists@intma.in>:
>> > On Wed, Apr 24, 2013 at 01:18:43PM +0200, Daniel Cegiełka wrote:
>> >>
>> >> btw. has anyone used go with musl?
>> >>
>> >
>> > Go ships its own libc, which I'm fairly certain it depends on.
>>
>> lib9, but are you sure about that?
>>
>> # ldd /usr/bin/go
>>       linux-vdso.so.1 =>  (0x00007fff50fff000)
>>       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff215e4b000)
>>       libc.so.6 => /lib64/libc.so.6 (0x00007ff215abf000)
>>       /lib64/ld-linux-x86-64.so.2 (0x00007ff216068000)
>
> Is the go binary written in go?

Good question! :) If so, it should be statically linked.

http://code.google.com/p/go/source/browse#hg%2Fsrc%2Fcmd%2Fgo

> The point was that go links programs
> against its own libc, not that the go compiler is linked against its
> own libc.

I'm not sure if they really use __only__ their own libc (lib9). In my
opinion lib9 seems to refer to libc.

Daniel

> Rich

Re: go support (was: Best place to discuss other lightweight libraries?)

[John Spencer]

On 04/24/2013 01:48 PM, Kurt H Maier wrote:
> On Wed, Apr 24, 2013 at 01:18:43PM +0200, Daniel Cegiełka wrote:
>>
>> btw. has anyone used go with musl?
>>
>
> Go ships its own libc, which I'm fairly certain it depends on.  It's
> also not suitable as a system programming language and they dropped that
> claim from their propaganda some time ago.

correct, the go runtime is *very* heavy, and it's always linked statically.
this adds ~ 1.5MB to any binary (at least on x86_64).
that's about equivalent to the bloat imposed by the C++ stdlib.

on the suckless page, there's something written about plans to migrate 
the coreutils functionality to go, this seems like an insane plan if 
even dead-simple tools like cat will eat 1.5 MB of your RAM and storage 
space.

Re: Re: go support (was: Best place to discuss other lightweight libraries?)

[Rich Felker]

On Wed, Apr 24, 2013 at 03:37:36PM +0200, John Spencer wrote:
> On 04/24/2013 01:48 PM, Kurt H Maier wrote:
> >On Wed, Apr 24, 2013 at 01:18:43PM +0200, Daniel Cegiełka wrote:
> >>
> >>btw. has anyone used go with musl?
> >>
> >
> >Go ships its own libc, which I'm fairly certain it depends on.  It's
> >also not suitable as a system programming language and they dropped that
> >claim from their propaganda some time ago.
> 
> correct, the go runtime is *very* heavy, and it's always linked statically.
> this adds ~ 1.5MB to any binary (at least on x86_64).
> that's about equivalent to the bloat imposed by the C++ stdlib.
> 
> on the suckless page, there's something written about plans to
> migrate the coreutils functionality to go, this seems like an insane
> plan if even dead-simple tools like cat will eat 1.5 MB of your RAM
> and storage space.

Storage space, yes. RAM, no. It's read-only mapping so it doesn't
consume commit charge, and it won't consume physical memory either
except for the parts that get paged in. In reality it might still use
less RAM than a dynamic-linked program, especially one linked with
glibc or with multiple shared libraries.

Rich

Re: Re: go support (was: Best place to discuss other lightweight libraries?)

[Kurt H Maier]

On Wed, Apr 24, 2013 at 03:37:36PM +0200, John Spencer wrote:
> 
> correct, the go runtime is *very* heavy, and it's always linked statically.
> this adds ~ 1.5MB to any binary (at least on x86_64).
> that's about equivalent to the bloat imposed by the C++ stdlib.
> 
> on the suckless page, there's something written about plans to migrate 
> the coreutils functionality to go, this seems like an insane plan if 
> even dead-simple tools like cat will eat 1.5 MB of your RAM and storage 
> space.
> 

Without getting into why none of that is true, the primary problem with
go as a systems language is the memory management.

khm

Re: Best place to discuss other lightweight libraries?

[Szabolcs Nagy]

* Kurt H Maier <khm-lists@intma.in> [2013-04-24 07:48:52 -0400]:
> On Wed, Apr 24, 2013 at 01:18:43PM +0200, Daniel Cegie??ka wrote:
> > 
> > btw. has anyone used go with musl?
> > 
> 
> Go ships its own libc, which I'm fairly certain it depends on.  It's
> also not suitable as a system programming language and they dropped that
> claim from their propaganda some time ago.
> 

go has its own independent world (own toolchain, syscall wrappers,
runtime, calling convention, stack management etc) but it can interact
with libc through cgo

so the question might be if anyone has tried cgo with musl
and i guess nobody tried but it should work since cgo does
not make much assumptions about the c runtime

go is special in this respect, most other language runtime
implementations build on top of libc so the interaction
between c and said language is less trivial

(there are some caveats in go as well: it does not call
__libc_start_main on startup nor exit on exit so eg atexit
handlers wont get called)

Re: Best place to discuss other lightweight libraries?

[Rich Felker]

On Wed, Apr 24, 2013 at 05:47:26PM +0200, Szabolcs Nagy wrote:
> * Kurt H Maier <khm-lists@intma.in> [2013-04-24 07:48:52 -0400]:
> > On Wed, Apr 24, 2013 at 01:18:43PM +0200, Daniel Cegie??ka wrote:
> > > 
> > > btw. has anyone used go with musl?
> > > 
> > 
> > Go ships its own libc, which I'm fairly certain it depends on.  It's
> > also not suitable as a system programming language and they dropped that
> > claim from their propaganda some time ago.
> > 
> 
> go has its own independent world (own toolchain, syscall wrappers,
> runtime, calling convention, stack management etc) but it can interact
> with libc through cgo
> 
> so the question might be if anyone has tried cgo with musl
> and i guess nobody tried but it should work since cgo does
> not make much assumptions about the c runtime
> 
> go is special in this respect, most other language runtime
> implementations build on top of libc so the interaction
> between c and said language is less trivial
> 
> (there are some caveats in go as well: it does not call
> __libc_start_main on startup nor exit on exit so eg atexit
> handlers wont get called)

The idea of calling functions in libc without __libc_start_main ever
having been called sounds highly misguided and potentially dangerous.
In musl it might mostly work, but with glibc I don't see how it could
possibly work.

Rich

Re: Best place to discuss other lightweight libraries?

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2013-04-24 15:17:35 -0400]:
> On Wed, Apr 24, 2013 at 05:47:26PM +0200, Szabolcs Nagy wrote:
> > (there are some caveats in go as well: it does not call
> > __libc_start_main on startup nor exit on exit so eg atexit
> > handlers wont get called)
> 
> The idea of calling functions in libc without __libc_start_main ever
> having been called sounds highly misguided and potentially dangerous.
> In musl it might mostly work, but with glibc I don't see how it could
> possibly work.

it works because _init is called by the loader before the entry point
and that does enough setup

(when cgo is used then the interpreter is set to /lib/ld-linux.so.2
in the elf header and it calls _init when it loads libc which is
listed in the dynamic section

the entry point is the go runtime which checks if it is in cgo mode
and sets up a separate pthread with libc managed stack and runs c
code there

the elf binary is prepared by the go toolchain, the go world is
statically linked including the wrapper, but there is an object
in the wrapper that is compiled with gcc and references the
extern symbols and uses the c abi, with a trick the go toolchain
knows about all the extern symbols and needed libraries which get
into the dynamic section so the c world is dynamically linked

there are some other runtime details to allow calling from go
into c and then calling back to go from c, bridging the difference
in abi and calling convenion, but basically this is how cgo works)

Re: Best place to discuss other lightweight libraries?

[Rob Landley]

On 04/24/2013 06:48:52 AM, Kurt H Maier wrote:
> On Wed, Apr 24, 2013 at 01:18:43PM +0200, Daniel Cegiełka wrote:
> >
> > btw. has anyone used go with musl?
> >
> 
> Go ships its own libc, which I'm fairly certain it depends on.  It's
> also not suitable as a system programming language and they dropped  
> that
> claim from their propaganda some time ago.

Ken Thompson is designing go, but Dennis Ritchie was really the guy who  
made C into a systems language (and kept pestering Ken to try doing  
bits of unix kernel in his latest version, then took the resulting  
feedback and implemented solutions for it).

For the curious, here's the paper from Dennis Ritchie's HOPL II talk:

   http://cm.bell-labs.com/cm/cs/who/dmr/chist.html

Rob

Re: go support (was: Best place to discuss other lightweight libraries?)

[John Spencer]

On 04/24/2013 01:18 PM, Daniel Cegiełka wrote:
>
> btw. has anyone used go with musl?
>

i tried to build gcc 4.7.2 with go support (--enable-languages=c,c++,go)
and that fails due to a lack of set/getcontext().
(see pkg/gcc472 in sabotage)

according to rich, adding that to musl requires a non-trivial amount of 
arch specific asm.

the go runtime in the gcc tree should be fixed to have a fallback when 
this functionality is missing (if possible),
so it maybe be needed to ask on the go mailing list.

Re: Re: go support (was: Best place to discuss other lightweight libraries?)

[Rich Felker]

On Wed, Apr 24, 2013 at 03:28:48PM +0200, John Spencer wrote:
> On 04/24/2013 01:18 PM, Daniel Cegiełka wrote:
> >
> >btw. has anyone used go with musl?
> >
> 
> i tried to build gcc 4.7.2 with go support (--enable-languages=c,c++,go)
> and that fails due to a lack of set/getcontext().
> (see pkg/gcc472 in sabotage)
> 
> according to rich, adding that to musl requires a non-trivial amount
> of arch specific asm.

Yes, but it is a wanted feature, so I wouldn't mind it getting done.
It was even part of the standard prior to POSIX 2008, and the reason
for removing it was stupid. (The reason was that the makecontext
function's calling convention is bogus and impossible to support
properly, but they could have fixed this by deprecating the use of the
variadic arguments in any way except passing a single void* argument,
rather than deprecating the whole set of interfaces.)

> the go runtime in the gcc tree should be fixed to have a fallback
> when this functionality is missing (if possible),
> so it maybe be needed to ask on the go mailing list.

The only fallback is to use C11 or POSIX threads in place of
coroutines, or shipping their own set/getcontext code for each arch...

Rich

Re: Best place to discuss other lightweight libraries?

[Christian Neukirchen]

Daniel Cegiełka <daniel.cegielka@gmail.com> writes:

> 2013/4/23 Luca Barbato <lu_zero@gentoo.org>:
>> On 04/23/2013 04:14 AM, Rob Landley wrote:
>>> What new crop of system languages?
>>
>> I could point Go and Rust as two that are trying too much for my taste.
>
> rust - ok, but written in c++; llvm as dependency.
>
> go - nice, only ed as a dependency but really stupid build system:

If "stupid" means "not clever", yes.  That's a feature.

-- 
Christian Neukirchen  <chneukirchen@gmail.com>  http://chneukirchen.org

Re: Best place to discuss other lightweight libraries?

[Daniel Cegiełka]

2013/4/24 Christian Neukirchen <chneukirchen@gmail.com>:
> Daniel Cegiełka <daniel.cegielka@gmail.com> writes:


>> go - nice, only ed as a dependency but really stupid build system:
>
> If "stupid" means "not clever", yes.  That's a feature.

it's more then stupid.

1) __hardcoded__ glibc loader (!!!)

https://code.google.com/p/go/source/browse/src/cmd/6l/asm.c
https://code.google.com/p/go/source/browse/src/cmd/8l/asm.c

you can run go (force musl libc) with command:

/lib/libc.so /bin/go (...)

2) hardcoded bison (instead ${YACC}):

https://code.google.com/p/go/source/browse/src/cmd/gc/Makefile

3) hardcoded bash in build scripts (instead portable sh).

4) C based 'build script' instead Makefile.

btw. Go (Google) works with musl (sabotage).

Daniel


> --
> Christian Neukirchen  <chneukirchen@gmail.com>  http://chneukirchen.org

Re: Go (was: [musl] Best place to discuss other lightweight libraries?)

[John Spencer]

On 04/29/2013 01:41 PM, Daniel Cegiełka wrote:

> btw. Go (Google) works with musl (sabotage).

oh ? which one ? the one that comes with gcc 4.7.2 doesn't.

Re: Re: Go (was: [musl] Best place to discuss other lightweight libraries?)

[Daniel Cegiełka]

2013/4/29 John Spencer <maillist-musl@barfooze.de>:
> On 04/29/2013 01:41 PM, Daniel Cegiełka wrote:
>
>> btw. Go (Google) works with musl (sabotage).
>
>
> oh ? which one ? the one that comes with gcc 4.7.2 doesn't.

http://code.google.com/p/go/

http://code.google.com/p/go/downloads/detail?name=go1.0.3.src.tar.gz&can=2&q=

But I'll have to review the code to clean up the hardcoded glibc
loader, bash, bison etc.

Best regards,
Daniel

Re: Best place to discuss other lightweight libraries?

[Rob Landley]

On 04/22/2013 04:52:48 PM, Rich Felker wrote:
> On Mon, Apr 22, 2013 at 05:21:25PM +0200, Luca Barbato wrote:
> > On 04/22/2013 04:53 PM, Rich Felker wrote:
> > > - Thread allergies, i.e. horribly over-complicating program logic  
> to
...
> zlib, etc.). Then, your main event loop just sees a normal file
> descriptor, so you don't have to invade your code that handles
> streams/connections/whatever with a new abstraction framework around
> file descriptors that also supports SSL or whatever. Not only are such
> abstraction layers bloated; they also add a lot of places for bugs to
...
> possible (e.g. a webserver). But for something like an IRC client or
> web browser, where socket performance is irrelevant compared to other
> things, it's idiotic to design fancy abstraction layers for reading
> and writing to connections when you could just do a design like what I
> described above.

If I might suggest: the general problem is unnecessary abstraction  
layers that hide the details of what you're doing from _yourself_, and  
which perform unnecessary work translating between your program and  
itself. (Let's marshall data into structures! Let's copy it back out  
again into _another_ structure! Let's do that five times in the same  
darn call chain! Bonus points for passing a hardwired constant as the  
only caller of a function that then checks that input for illegal  
values!)

It's nice to pull in a packaged solution that automates away fiddly  
bits and tedium with tested and working code (which is basically what  
libc is), but pulling in 8 megabytes of shared library to beep the  
speaker instead of writing to a /proc file? Very common, and not an  
improvement. (Random current example: why does qemu need an extended  
attribute library to build the virtfs server? It doesn't pull in a  
libchmod.so to set file ownership and permissions...)

> > > - DBus.
> >
> > Sadly nobody is pushing for a better local socket multicast  
> abstraction
> > to send notifications back and forth in an efficient fashion.

When I say that it always winds up being my job. (Belling the cat  
protocol.)

> > I'm hoping for nanomsg once it is complete or Binder once it is
> > correctly documented ^^; (and thus implemented in more than few  
> forks of
> > linux and maybe haiku)
> 
> Yes, Binder looks really promising, but I also think part of the
> problem is that the need for this kind of interface is exaggerated...

Is anybody collecting patches to remove it from packages that currently  
require it? If it's easy to Not Do That, it should be possible to  
demonstrate...

> > > - Use of global state. Even seemingly-harmless things like a  
> global
> > >   registered log function are harmful, because two different  
> libraries
> > >   (or the main program and a library) might be trying to use the
> > >   library with the global log destination, and clobbering each  
> other's
> > >   choices.
> >
> > For this there aren't solution that won't cause different problems  
> I'm
> > afraid.
> 
> Sure there are. I get the impression you can tell I was talking about
> libav/ffmpeg's log interface. :-) The obvious solution is to bind log
> contexts to the object you're acting on. See this nice talk:
> 
> http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/

Was that the java one?

> If I remember right, part of the problem way back was that there were
> deep function calls that had no context available to them, and that
> didn't _need_ a context for anything but logging warnings or whatnot.
> Really, the fact that they can fail and want to be able to report
> failure means they _do_ need a context, but I can understand the
> desire to cheat, especially if there's a performance cost to passing
> the context all the way down. In that case, hidden non-global state,
> namely thread-local storage, might be an appropriate solution. It's
> still hidden state (and thus A Bad Thing), but at least it's no longer
> global state.

There are contexts between global and local. There have always been  
contexts between global and local. This is why objects were invented,  
to provide a better syntax for contexts between global and local.

But another issue is visibility of contexts. For example, if you put  
stuff in environment variables, you can view and edit the environment  
variables. So if you want it to stop doing something, or see whether or  
not it's doing something, it's easy to dump state with any number of  
tools.

Unix had reasonable solutions for this stuff in the 1970's. Java  
reinvented the wheel and thought "hexagonal" was a good idea, and it  
_sort_ of works, and they've been filing bits off and gluing other bits  
on ever since. (Alas, mostly they've just been making their wheel  
_thicker_. Oh well.)

Rob