Priorities for next release?

Priorities for next release?

[Rich Felker]

Hi all,
Since we have some major progress since 0.9.3, I'm thinking of making
the next release pretty soon. Big things so far:

- Blowfish crypt
- MIPS dynamic linker
- Major MIPS bug fixes
- ARM hard float support

Among the other improvements that are pending, what else would you
like to see in 0.9.4? I know there's a pending bug fix for wcsstr (and
perhaps the wcs* functions could use some checking in general). I have
asm for memcpy on i386 and x86_64 which could be committed. It might
also be desirable to get md5 and sha hash functions for crypt
integrated if anybody can provide efficient ones under appropriate
license. My understanding is that they're mainly for compatibility
with existing password hashes and otherwise inferrior to blowfish, so
I'd really like to focus on having them be small and unobtrusive at
the expense of some speed.

Other requests?

Rich

Re: Priorities for next release?

[Nathan McSween]

On 8/10/2012 12:12 PM, Rich Felker wrote:
> Hi all,
> Since we have some major progress since 0.9.3, I'm thinking of making
> the next release pretty soon. Big things so far:
>
> - Blowfish crypt
> - MIPS dynamic linker
> - Major MIPS bug fixes
> - ARM hard float support
>
> Among the other improvements that are pending, what else would you
> like to see in 0.9.4? I know there's a pending bug fix for wcsstr (and
> perhaps the wcs* functions could use some checking in general). I have
> asm for memcpy on i386 and x86_64 which could be committed. It might
> also be desirable to get md5 and sha hash functions for crypt
> integrated if anybody can provide efficient ones under appropriate
> license. My understanding is that they're mainly for compatibility
> with existing password hashes and otherwise inferrior to blowfish, so
> I'd really like to focus on having them be small and unobtrusive at
> the expense of some speed.
>
> Other requests?
>
> Rich
Unit tests of some sort, it helps to document the code as well as stop 
some regressions.

Unit/regression testing (was Re: [musl] Priorities for next release?)

[Rich Felker]

On Fri, Aug 10, 2012 at 12:23:17PM -0700, Nathan McSween wrote:
> >Other requests?
> 
> Unit tests of some sort, it helps to document the code as well as
> stop some regressions.

There are already a number of different tests; see:

http://nsz.repo.hu/git/
http://git.etalabs.net/cgi-bin/cgit.cgi
https://github.com/lmarcetic/cluts

for some that were developed specifically with musl in mind. We've
also been using:

http://posixtest.sourceforge.net/

but it's rather outdated and incorrect in some areas.

I've intentionally kept tests out of the main musl source tree for two
reasons: For one thing, I want to avoid burdening non-developers who
just want to download musl to use it with unnecessary download/storage
bulk. But the other reason is that I want the majority of libc test
cases to be libc-agnostic, i.e. designed to run against any libc, not
just musl. In that case, including them as part of musl feels wrong. I
do however also want to develop some musl-specific regression tests
for things that can't be tested in a generic way: things like ensuring
that static linking doesn't pull in modules it shouldn't, that
unnecessary syscall overhead at startup doesn't creep in, or that
musl-specific QoI guarantees are preserved (such as certain interfaces
not taking superlinear time or being async-signal-safe even when not
required by POSIX).

As to how testing relates to this release thread, I think development
of further major tests is outside the realm of the immediate sort of
release priorities I was asking about. But it's something I've been
wanting to pursue again for a long time, and I'd be quite happy to
find contributors interested in working on it.

Rich

Re: Priorities for next release?

[Luca Barbato]

On 8/10/12 9:12 PM, Rich Felker wrote:
> Hi all,
> Since we have some major progress since 0.9.3, I'm thinking of making
> the next release pretty soon. Big things so far:
>
> - Blowfish crypt
> - MIPS dynamic linker
> - Major MIPS bug fixes
> - ARM hard float support
>
> Among the other improvements that are pending, what else would you
> like to see in 0.9.4?

As annoying as it is XOPEN ifdeffery would be a boon...

lu

Re: Priorities for next release?

[Rich Felker]

On Sat, Aug 11, 2012 at 12:03:45AM +0200, Luca Barbato wrote:
> On 8/10/12 9:12 PM, Rich Felker wrote:
> >Hi all,
> >Since we have some major progress since 0.9.3, I'm thinking of making
> >the next release pretty soon. Big things so far:
> >
> >- Blowfish crypt
> >- MIPS dynamic linker
> >- Major MIPS bug fixes
> >- ARM hard float support
> >
> >Among the other improvements that are pending, what else would you
> >like to see in 0.9.4?
> 
> As annoying as it is XOPEN ifdeffery would be a boon...

What do you mean?

Rich

Re: Priorities for next release?

[Luca Barbato]

On 8/11/12 2:05 AM, Rich Felker wrote:
> On Sat, Aug 11, 2012 at 12:03:45AM +0200, Luca Barbato wrote:
>> On 8/10/12 9:12 PM, Rich Felker wrote:
>>> Hi all,
>>> Since we have some major progress since 0.9.3, I'm thinking of making
>>> the next release pretty soon. Big things so far:
>>>
>>> - Blowfish crypt
>>> - MIPS dynamic linker
>>> - Major MIPS bug fixes
>>> - ARM hard float support
>>>
>>> Among the other improvements that are pending, what else would you
>>> like to see in 0.9.4?
>>
>> As annoying as it is XOPEN ifdeffery would be a boon...
>
> What do you mean?

-D_XOPEN_SOURCE=600 and such.

Re: Priorities for next release?

[Rich Felker]

On Sat, Aug 11, 2012 at 02:54:25AM +0200, Luca Barbato wrote:
> On 8/11/12 2:05 AM, Rich Felker wrote:
> >On Sat, Aug 11, 2012 at 12:03:45AM +0200, Luca Barbato wrote:
> >>On 8/10/12 9:12 PM, Rich Felker wrote:
> >>>Hi all,
> >>>Since we have some major progress since 0.9.3, I'm thinking of making
> >>>the next release pretty soon. Big things so far:
> >>>
> >>>- Blowfish crypt
> >>>- MIPS dynamic linker
> >>>- Major MIPS bug fixes
> >>>- ARM hard float support
> >>>
> >>>Among the other improvements that are pending, what else would you
> >>>like to see in 0.9.4?
> >>
> >>As annoying as it is XOPEN ifdeffery would be a boon...
> >
> >What do you mean?
> 
> -D_XOPEN_SOURCE=600 and such.

What do you mean? You want support for exposing SUSv3 symbols but
hiding things that were added in SUSv4? Is there any particular
reason?

Rich

Re: Priorities for next release?

[idunham]

> On 8/11/12 2:05 AM, Rich Felker wrote:
>> On Sat, Aug 11, 2012 at 12:03:45AM +0200, Luca Barbato wrote:
>>> On 8/10/12 9:12 PM, Rich Felker wrote:
>>>> Hi all,
>>>> Since we have some major progress since 0.9.3, I'm thinking of making
>>>> the next release pretty soon. Big things so far:
>>>>
>>>> - Blowfish crypt
>>>> - MIPS dynamic linker
>>>> - Major MIPS bug fixes
>>>> - ARM hard float support
all very good...
Out of curiousity, have you (or anyone else, for that matter) checked how
much of the LSB ABI is supported as of yet?

Also, if we're talking about adding MD5 or SHA (of the two, I think sha is
the more important) along the line, I'd think that adding multiple
crypt-related APIs in one release makes sense. Not all that important,
though.

I'm guessing that merging ppc/mips64/microblaze ports will end up waiting
for another release?

>>>> Among the other improvements that are pending, what else would you
>>>> like to see in 0.9.4?
>>>
>>> As annoying as it is XOPEN ifdeffery would be a boon...
>>
>> What do you mean?
>
> -D_XOPEN_SOURCE=600 and such.

Points to consider:
1-it might slow down compiling and make headers ugly.
2-as the person who added _BSD_SOURCE, I can assure you that it's not a
quick project. And with school starting in 2 weeks, I'm not the one doing
it.
(I don't think it's going to happen timely enough for the next release,
certainly)
3- -D_GNU_SOURCE will do for building most of this stuff
4-if you want a header change, may I politely suggest that you do the
patches?

HTH,
Isaac Dunham

Re: Priorities for next release?

[Rich Felker]

On Fri, Aug 10, 2012 at 10:21:21PM -0400, idunham@lavabit.com wrote:
> >>>> - Blowfish crypt
> >>>> - MIPS dynamic linker
> >>>> - Major MIPS bug fixes
> >>>> - ARM hard float support
> all very good...
> Out of curiousity, have you (or anyone else, for that matter) checked how
> much of the LSB ABI is supported as of yet?

No, unfortunately the published LSB is fairly outdated, and glibc
programs built with special options like certain inlining options or
-D_FORTIFY_SOURCE depend on symbols that are not in LSB (as published)
even if they don't use any nonstandard interfaces at the source
level..

> Also, if we're talking about adding MD5 or SHA (of the two, I think sha is
> the more important) along the line, I'd think that adding multiple
> crypt-related APIs in one release makes sense. Not all that important,
> though.

Yes, I tend to agree, both that SHA is more important and that it
would be nice to get them all added in one release.

> I'm guessing that merging ppc/mips64/microblaze ports will end up waiting
> for another release?

Depends on how inspired I get.. ;-)
An experimental merge could definitely happen before a release if I
jump into it. Having something well-tested and promoted as
complete/working is not going to happen.

> >>>> Among the other improvements that are pending, what else would you
> >>>> like to see in 0.9.4?
> >>>
> >>> As annoying as it is XOPEN ifdeffery would be a boon...
> >>
> >> What do you mean?
> >
> > -D_XOPEN_SOURCE=600 and such.
> 
> Points to consider:
> 1-it might slow down compiling and make headers ugly.
> 2-as the person who added _BSD_SOURCE, I can assure you that it's not a
> quick project. And with school starting in 2 weeks, I'm not the one doing
> it.
> (I don't think it's going to happen timely enough for the next release,
> certainly)
> 3- -D_GNU_SOURCE will do for building most of this stuff
> 4-if you want a header change, may I politely suggest that you do the
> patches?

I agree with all these points. I think trying to support multiple
versions of the standards adds a lot of complexity with little
tangible benefit.

One variation that might be useful however is adding things like

#if defined(_XOPEN_SOURCE) && _XOPEN_SOURCE <= 600

as a condition around things that were removed in SUSv4 but previously
widely used, like gethostbyname and usleep. Of course the
corresponding tests for _POSIX_C_SOURCE would also belong there, as
appropriate.

Making -D_XOPEN_SOURCE=600 hide the new stuff from SUSv4 would just be
gratuitous ugliness in my mind, but making it re-expose stuff that was
removed later sounds potentially useful.

Rich

Re: Priorities for next release?

[Daniel Cegiełka]

support for:

extern char *__progname;

Daniel

Re: Priorities for next release?

[Rich Felker]

On Sat, Aug 11, 2012 at 02:21:39PM +0200, Daniel Cegiełka wrote:
> support for:
> 
> extern char *__progname;

Isn't there a different public name for this? Policy is generally not
to reproduce symbols that begin with __ unless they are:

(1) part of a documented public API, like stdio_ext.h, or
(2) not used as an API, but part of the ABI that results from
compiling a program against glibc headers.

This is the same issue as __secure_getenv, and the glibc folks
themselves even have declared that it's not to be used and made a
replacement symbol for the public interface.

Rich

Re: Priorities for next release?

[Daniel Cegiełka]

2012/8/11 Rich Felker <dalias@aerifal.cx>:
> On Sat, Aug 11, 2012 at 02:21:39PM +0200, Daniel Cegiełka wrote:
>> support for:
>>
>> extern char *__progname;
>
> Isn't there a different public name for this?

argv[0]

http://fossies.org/dox/glibc-2.16.0/init-misc_8c_source.html

Unfortunately __progname is required by many packages from *BSD...

Daniel

Re: Priorities for next release?

[Rich Felker]

On Sat, Aug 11, 2012 at 02:35:52PM +0200, Daniel Cegiełka wrote:
> 2012/8/11 Rich Felker <dalias@aerifal.cx>:
> > On Sat, Aug 11, 2012 at 02:21:39PM +0200, Daniel Cegiełka wrote:
> >> support for:
> >>
> >> extern char *__progname;
> >
> > Isn't there a different public name for this?
> 
> argv[0]

I meant program_invocation_name or whatever ugly thing glibc calls it.
Of course if the goal is to make BSD programs happy, that won't really
help. I really wish somebody would just fix these programs to save
argv[0] themselves if they need it (or to hard-code their own names as
a string literal...) rather than keeping around the historical
practice of poking at implementation internals of the original libc
they used..

> http://fossies.org/dox/glibc-2.16.0/init-misc_8c_source.html

Yes, that's the "proper" public names...

Rich

Re: Priorities for next release?

[Daniel Cegiełka]

>> argv[0]
>
> I meant program_invocation_name or whatever ugly thing glibc calls it.
> Of course if the goal is to make BSD programs happy, that won't really
> help. I really wish somebody would just fix these programs to save
> argv[0] themselves if they need it (or to hard-code their own names as
> a string literal...) rather than keeping around the historical
> practice of poking at implementation internals of the original libc
> they used..

That's what I do now (patches for __progname). Unfortunately some
linux distributions forcing __progname, even Owl:

in Owl's popa3d we have a #ifdef, which really helps.

#ifdef HAVE_PROGNAME
extern char *__progname;
#define progname __progname

http://cvsweb.openwall.com/cgi/cvsweb.cgi/Owl/packages/popa3d/popa3d/startup.c?rev=1.4;content-type=text%2Fplain

but for Owl's msulogin we need a patch for musl:

http://cvsweb.openwall.com/cgi/cvsweb.cgi/Owl/packages/msulogin/msulogin/sulogin.c?rev=1.4;content-type=text%2Fplain

Daniel

Re: Priorities for next release?

[Rich Felker]

On Sat, Aug 11, 2012 at 06:35:12PM +0200, Daniel Cegiełka wrote:
> >> argv[0]
> >
> > I meant program_invocation_name or whatever ugly thing glibc calls it.
> > Of course if the goal is to make BSD programs happy, that won't really
> > help. I really wish somebody would just fix these programs to save
> > argv[0] themselves if they need it (or to hard-code their own names as
> > a string literal...) rather than keeping around the historical
> > practice of poking at implementation internals of the original libc
> > they used..
> 
> That's what I do now (patches for __progname). Unfortunately some
> linux distributions forcing __progname, even Owl:
> 
> in Owl's popa3d we have a #ifdef, which really helps.
> 
> #ifdef HAVE_PROGNAME
> extern char *__progname;
> #define progname __progname
> 
> http://cvsweb.openwall.com/cgi/cvsweb.cgi/Owl/packages/popa3d/popa3d/startup.c?rev=1.4;content-type=text%2Fplain
> 
> but for Owl's msulogin we need a patch for musl:
> 
> http://cvsweb.openwall.com/cgi/cvsweb.cgi/Owl/packages/msulogin/msulogin/sulogin.c?rev=1.4;content-type=text%2Fplain

Couldn't this code just be fixed to add an argument to usage() and
call usage(argv[0]) or even usage(basename(argv[0])) if preferred?

Rich

Re: Priorities for next release?

[orc]

On Sat, 11 Aug 2012 12:50:49 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> On Sat, Aug 11, 2012 at 06:35:12PM +0200, Daniel Cegiełka wrote:
> > >> argv[0]
> > >
> > > I meant program_invocation_name or whatever ugly thing glibc
> > > calls it. Of course if the goal is to make BSD programs happy,
> > > that won't really help. I really wish somebody would just fix
> > > these programs to save argv[0] themselves if they need it (or to
> > > hard-code their own names as a string literal...) rather than
> > > keeping around the historical practice of poking at
> > > implementation internals of the original libc they used..
> > 
> > That's what I do now (patches for __progname). Unfortunately some
> > linux distributions forcing __progname, even Owl:
> > 
> > in Owl's popa3d we have a #ifdef, which really helps.
> > 
> > #ifdef HAVE_PROGNAME
> > extern char *__progname;
> > #define progname __progname
> > 
> > http://cvsweb.openwall.com/cgi/cvsweb.cgi/Owl/packages/popa3d/popa3d/startup.c?rev=1.4;content-type=text%2Fplain
> > 
> > but for Owl's msulogin we need a patch for musl:
> > 
> > http://cvsweb.openwall.com/cgi/cvsweb.cgi/Owl/packages/msulogin/msulogin/sulogin.c?rev=1.4;content-type=text%2Fplain
> 
> Couldn't this code just be fixed to add an argument to usage() and
> call usage(argv[0]) or even usage(basename(argv[0])) if preferred?
> 
> Rich

Or even progname = basename(argv[0]) with global char *progname ?

Re: Priorities for next release?

[Daniel Cegiełka]

>> Couldn't this code just be fixed to add an argument to usage() and
>> call usage(argv[0]) or even usage(basename(argv[0])) if preferred?
>>
>> Rich
>
> Or even progname = basename(argv[0]) with global char *progname ?

Usually it looks like this:

extern char *__progname;

'extern' means __progname must be available to external objects, so in
code with main():

#include <stdio.h>

char *__progname;

int main(int argc, char *argv[])
{
        __progname = argv[0];
        printf("%s\n", __progname);
        return 0;
}

what you think? It works for me... so why basename(argv[0])?

Daniel

Re: Priorities for next release?

[orc]

On Sat, 11 Aug 2012 20:41:57 +0200
Daniel Cegiełka <daniel.cegielka@gmail.com> wrote:

> >> Couldn't this code just be fixed to add an argument to usage() and
> >> call usage(argv[0]) or even usage(basename(argv[0])) if preferred?
> >>
> >> Rich
> >
> > Or even progname = basename(argv[0]) with global char *progname ?
> 
> Usually it looks like this:
> 
> extern char *__progname;
> 
> 'extern' means __progname must be available to external objects, so in
> code with main():
> 
> #include <stdio.h>
> 
> char *__progname;
> 
> int main(int argc, char *argv[])
> {
>         __progname = argv[0];
>         printf("%s\n", __progname);
>         return 0;
> }
> 
> what you think? It works for me... so why basename(argv[0])?
> 
> Daniel

basename() is for stripping path objects, for example, when you run
prog via full path, /bin/true, basename("/bin/true") = "true"

See man 3 basename

Re: Priorities for next release?

[Daniel Cegiełka]

>> what you think? It works for me... so why basename(argv[0])?
>>
>> Daniel
>
> basename() is for stripping path objects, for example, when you run
> prog via full path, /bin/true, basename("/bin/true") = "true"
>
> See man 3 basename

Indeed, it is better. However, if you run from $PATH, it works ok
without basename().

Thanks for suggestion :)

Daniel

Re: Priorities for next release?

[Rich Felker]

On Sat, Aug 11, 2012 at 09:05:41PM +0200, Daniel Cegiełka wrote:
> >> what you think? It works for me... so why basename(argv[0])?
> >>
> >> Daniel
> >
> > basename() is for stripping path objects, for example, when you run
> > prog via full path, /bin/true, basename("/bin/true") = "true"
> >
> > See man 3 basename
> 
> Indeed, it is better. However, if you run from $PATH, it works ok
> without basename().

I was just trying to duplicate existing behavior, which is for
__progname to be the basename-only part. By the way, if patching the
program to have its own global var, it must not be named __progname
but progname or something else. __progname (beginning with 2
underscores) is in the namespace reserved for the implementation and
it's a bug for an application to define names in this space.

Rich

Re: Priorities for next release?

[Rich Felker]

On Fri, Aug 10, 2012 at 03:12:55PM -0400, Rich Felker wrote:
> Other requests?

Summary so far:

- significand - easy to do
- __progname - questionable whether it should be added
- _XOPEN_SOURCE versions - hard to do, also questionable
- unit tests - very desirable, not going to happen in short time
- md5/sha crypt - seems harder than i thought; can't find small code

Did I miss anything? Other additions to the wishlist for next release?

Rich

Re: Priorities for next release?

[Daniel Cegiełka]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

2012/8/11 Rich Felker <dalias@aerifal.cx>:
> On Fri, Aug 10, 2012 at 03:12:55PM -0400, Rich Felker wrote:
>> Other requests?
>
> Summary so far:
>
> - significand - easy to do
> - __progname - questionable whether it should be added
> - _XOPEN_SOURCE versions - hard to do, also questionable
> - unit tests - very desirable, not going to happen in short time
> - md5/sha crypt - seems harder than i thought; can't find small code
>
> Did I miss anything? Other additions to the wishlist for next release?

Support for the man pages? This can be simple dir in musl
(man/man{1,2,...) and "--with-man" option in configure.

I'm sending fgetln.c (+my diff), but please check it...
btw. it based on /usr.bin/make/util.c from OpenBSD:

http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/make/util.c?rev=1.26;content-type=text%2Fplain

Daniel


> Rich

[-- Attachment #2: openbsd_fgetln.diff --]
[-- Type: application/octet-stream, Size: 6255 bytes --]

diff -urN musl.orig//include/stdio.h musl/include/stdio.h
--- musl.orig//include/stdio.h	2012-08-11 18:02:58.000000000 +0200
+++ musl/include/stdio.h	2012-08-11 18:07:38.522807627 +0200
@@ -92,6 +92,10 @@
 char *fgets(char *, int, FILE *);
 char *gets(char *);
 
+#ifdef _BSD_SOURCE
+char *fgetln(FILE *, size_t *);
+#endif
+
 int fputs(const char *, FILE *);
 int puts(const char *);
 
diff -urN musl.orig//src/stdio/fgetln.3 musl/src/stdio/fgetln.3
--- musl.orig//src/stdio/fgetln.3	1970-01-01 01:00:00.000000000 +0100
+++ musl/src/stdio/fgetln.3	2007-06-21 15:47:07.000000000 +0200
@@ -0,0 +1,146 @@
+.\"	$OpenBSD: fgetln.3,v 1.15 2007/06/21 13:47:07 moritz Exp $
+.\"
+.\" Copyright (c) 1990, 1991, 1993
+.\"	The Regents of the University of California.  All rights reserved.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\" 3. Neither the name of the University nor the names of its contributors
+.\"    may be used to endorse or promote products derived from this software
+.\"    without specific prior written permission.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.Dd $Mdocdate: June 21 2007 $
+.Dt FGETLN 3
+.Os
+.Sh NAME
+.Nm fgetln
+.Nd get a line from a stream
+.Sh SYNOPSIS
+.Fd #include <stdio.h>
+.Ft char *
+.Fn fgetln "FILE *stream" "size_t *len"
+.Sh DESCRIPTION
+The
+.Fn fgetln
+function returns a pointer to the next line from the stream referenced by
+.Fa stream .
+This line is
+.Em not
+a C string as it does not end with a terminating
+.Tn NUL
+character.
+The length of the line, including the final newline,
+is stored in the memory location to which
+.Fa len
+points and is guaranteed to be greater than 0 upon successful completion.
+(Note, however, that if the last line in the stream does not end in a newline,
+the returned text will not contain a newline.)
+.Sh RETURN VALUES
+Upon successful completion a pointer is returned;
+this pointer becomes invalid after the next
+.Tn I/O
+operation on
+.Fa stream
+(whether successful or not)
+or as soon as the stream is closed.
+Otherwise,
+.Dv NULL
+is returned.
+.Pp
+The
+.Fn fgetln
+function does not distinguish between end-of-file and error; the routines
+.Xr feof 3
+and
+.Xr ferror 3
+must be used
+to determine which occurred.
+If an error occurs, the global variable
+.Va errno
+is set to indicate the error.
+The end-of-file condition is remembered, even on a terminal, and all
+subsequent attempts to read will return
+.Dv NULL
+until the condition is
+cleared with
+.Xr clearerr 3 .
+.Pp
+The text to which the returned pointer points may be modified,
+provided that no changes are made beyond the returned size.
+These changes are lost as soon as the pointer becomes invalid.
+.Sh ERRORS
+.Bl -tag -width [EBADF]
+.It Bq Er EBADF
+The argument
+.Fa stream
+is not a stream open for reading.
+.El
+.Pp
+The
+.Fn fgetln
+function may also fail and set
+.Va errno
+for any of the errors specified for the routines
+.Xr fflush 3 ,
+.Xr malloc 3 ,
+.Xr read 2 ,
+.Xr stat 2 ,
+or
+.Xr realloc 3 .
+.Sh SEE ALSO
+.Xr ferror 3 ,
+.Xr fgets 3 ,
+.Xr fopen 3 ,
+.Xr fparseln 3 ,
+.Xr putc 3
+.Sh HISTORY
+The
+.Fn fgetln
+function first appeared in
+.Bx 4.4 .
+.Sh CAVEATS
+Since the returned buffer is not a C string (it is not NUL terminated), a
+common practice is to replace the newline character with
+.Sq \e0 .
+However, if the last line in a file does not contain a newline,
+the returned text won't contain a newline either.
+The following code demonstrates how to deal with this problem by allocating a
+temporary buffer:
+.Bd -literal
+	char *buf, *lbuf;
+	size_t len;
+
+	lbuf = NULL;
+	while ((buf = fgetln(fp, &len))) {
+		if (buf[len - 1] == '\en')
+			buf[len - 1] = '\e0';
+		else {
+			/* EOF without EOL, copy and add the NUL */
+			if ((lbuf = malloc(len + 1)) == NULL)
+				err(1, NULL);
+			memcpy(lbuf, buf, len);
+			lbuf[len] = '\e0';
+			buf = lbuf;
+		}
+		printf("%s\en", buf);
+	}
+	free(lbuf);
+.Ed
diff -urN musl.orig//src/stdio/fgetln.c musl/src/stdio/fgetln.c
--- musl.orig//src/stdio/fgetln.c	1970-01-01 01:00:00.000000000 +0100
+++ musl/src/stdio/fgetln.c	2012-08-11 18:04:33.875929278 +0200
@@ -0,0 +1,38 @@
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <sys/types.h>
+
+char *
+fgetln(stream, len)
+    FILE *stream;
+    size_t *len;
+{
+    static char *buffer = NULL;
+    static size_t buflen = 0;
+
+    if (buflen == 0) {
+	buflen = 512;
+	if ((buffer = malloc(buflen+1)) == NULL) {
+		fprintf(stderr, "malloc: %s (%zu)\n", strerror(errno),
+			buflen+1);
+		exit(2);
+	}
+    }
+    if (fgets(buffer, buflen+1, stream) == NULL)
+	return NULL;
+    *len = strlen(buffer);
+    while (*len == buflen && buffer[*len-1] != '\n') {
+	if ((buffer = realloc(buffer, 2*buflen+1)) == NULL) {
+		fprintf(stderr, "realloc: %s (%zu)\n", strerror(errno),
+			2*buflen+1);
+		exit(2);
+	}
+	if (fgets(buffer + buflen, buflen + 1, stream) == NULL)
+	    return NULL;
+	*len += strlen(buffer + buflen);
+	buflen *= 2;
+    }
+    return buffer;
+}

[-- Attachment #3: fgetln.c --]
[-- Type: text/x-csrc, Size: 840 bytes --]

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>

char *
fgetln(stream, len)
    FILE *stream;
    size_t *len;
{
    static char *buffer = NULL;
    static size_t buflen = 0;

    if (buflen == 0) {
	buflen = 512;
	if ((buffer = malloc(buflen+1)) == NULL) {
		fprintf(stderr, "malloc: %s (%zu)\n", strerror(errno),
			buflen+1);
		exit(2);
	}
    }
    if (fgets(buffer, buflen+1, stream) == NULL)
	return NULL;
    *len = strlen(buffer);
    while (*len == buflen && buffer[*len-1] != '\n') {
	if ((buffer = realloc(buffer, 2*buflen+1)) == NULL) {
		fprintf(stderr, "realloc: %s (%zu)\n", strerror(errno),
			2*buflen+1);
		exit(2);
	}
	if (fgets(buffer + buflen, buflen + 1, stream) == NULL)
	    return NULL;
	*len += strlen(buffer + buflen);
	buflen *= 2;
    }
    return buffer;
}

Re: Priorities for next release?

[Rich Felker]

On Sat, Aug 11, 2012 at 10:26:35PM +0200, Daniel Cegiełka wrote:
> > Did I miss anything? Other additions to the wishlist for next release?
> 
> Support for the man pages? This can be simple dir in musl
> (man/man{1,2,...) and "--with-man" option in configure.

Hm? Adding man pages for every interface would be a huge task, well
outside of "finishing touches for the next release". :-) I'd actually
like to find someone interested in writing detailed documentation
(there was a thread about this a while back), but I'm not sure man
pages are the best format; the POSIX man pages suffice as programmers'
documentation for most of the interfaces in musl.

> I'm sending fgetln.c (+my diff), but please check it...
> btw. it based on /usr.bin/make/util.c from OpenBSD:

If we add fgetln, I'd like a much higher quality of implementation.
It's not clear from the past documentation I've read for this function
that it's allowed to use a shared static buffer for all FILEs, and
even if it were, I find that really ugly. Instead, simply returning a
pointer into the FILE's buffer when the whole line is already present
in the buffer, and otherwise allocating a FILE-local buffer for it,
would be a lot nicer. fclose could then check the FILE-local pointer
and free if it it was allocated.

I was under the impression that this was how legacy BSD fgetln worked
in the first place...

Rich

P.S. Just noticed another thing: K&R function arguments are not valid
C99 or C11 and are not okay in musl.

Re: Priorities for next release?

[Rich Felker]

On Sat, Aug 11, 2012 at 04:51:28PM -0400, Rich Felker wrote:
> > I'm sending fgetln.c (+my diff), but please check it...
> > btw. it based on /usr.bin/make/util.c from OpenBSD:
> 
> If we add fgetln, I'd like a much higher quality of implementation.
> It's not clear from the past documentation I've read for this function
> that it's allowed to use a shared static buffer for all FILEs, and
> even if it were, I find that really ugly. Instead, simply returning a

Indeed, even the man page you included states that the returned buffer
becomes invalid after the next I/O operation on stream, not after a
call to fgetln on a different stream. Thus I think the implementation
fails to conform to the contract documented in the man page..

Rich

Re: Priorities for next release?

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

On Sat, Aug 11, 2012 at 04:51:28PM -0400, Rich Felker wrote:
> > I'm sending fgetln.c (+my diff), but please check it...
> > btw. it based on /usr.bin/make/util.c from OpenBSD:
> 
> If we add fgetln, I'd like a much higher quality of implementation.
> It's not clear from the past documentation I've read for this function
> that it's allowed to use a shared static buffer for all FILEs, and
> even if it were, I find that really ugly. Instead, simply returning a
> pointer into the FILE's buffer when the whole line is already present
> in the buffer, and otherwise allocating a FILE-local buffer for it,
> would be a lot nicer. fclose could then check the FILE-local pointer
> and free if it it was allocated.

See attached. Does it work okay?

Rich

[-- Attachment #2: fgetln.c --]
[-- Type: text/plain, Size: 341 bytes --]

#include "stdio_impl.h"

char *fgetln(FILE *f, size_t *plen)
{
	char *z;
	ssize_t l;
	if ((z=memchr(f->rpos, '\n', f->rend - f->rpos))) {
		char *ret = (char *)f->rpos;
		*plen = ++z - ret;
		f->rpos = (void *)z;
		return ret;
	}
	if ((l = getline(&f->getln_buf, (size_t[]){0}, f)) > 0) {
		*plen = l;
		return f->getln_buf;
	}
	return 0;
}

[-- Attachment #3: fgetln.diff --]
[-- Type: text/plain, Size: 631 bytes --]

diff --git a/src/internal/stdio_impl.h b/src/internal/stdio_impl.h
index d54c918..65dcfbd 100644
--- a/src/internal/stdio_impl.h
+++ b/src/internal/stdio_impl.h
@@ -57,7 +57,7 @@ struct __FILE_s {
 	int waiters;
 	void *cookie;
 	off_t off;
-	void *dummy4;
+	char *getln_buf;
 	void *mustbezero_2;
 	unsigned char *shend;
 	off_t shlim, shcnt;
diff --git a/src/stdio/fclose.c b/src/stdio/fclose.c
index 373a2c7..8fdc3f7 100644
--- a/src/stdio/fclose.c
+++ b/src/stdio/fclose.c
@@ -16,6 +16,7 @@ int fclose(FILE *f)
 	r = fflush(f);
 	r |= f->close(f);
 
+	if (f->getln_buf) free(f->getln_buf);
 	if (!perm) free(f);
 	
 	return r;

Re: Priorities for next release?

[Daniel Cegiełka]

2012/8/11 Rich Felker <dalias@aerifal.cx>:
> On Sat, Aug 11, 2012 at 04:51:28PM -0400, Rich Felker wrote:
>> > I'm sending fgetln.c (+my diff), but please check it...
>> > btw. it based on /usr.bin/make/util.c from OpenBSD:
>>
>> If we add fgetln, I'd like a much higher quality of implementation.
>> It's not clear from the past documentation I've read for this function
>> that it's allowed to use a shared static buffer for all FILEs, and
>> even if it were, I find that really ugly. Instead, simply returning a
>> pointer into the FILE's buffer when the whole line is already present
>> in the buffer, and otherwise allocating a FILE-local buffer for it,
>> would be a lot nicer. fclose could then check the FILE-local pointer
>> and free if it it was allocated.
>
> See attached. Does it work okay?
>
> Rich

Thank you! And I see that is already in the repo... so I built a new
version (with new memcpy).

btw. I see that musl supports getdents(), but it is not visible in dirent.h.

Daniel

Re: Priorities for next release?

[Rich Felker]

On Sat, Aug 11, 2012 at 03:56:49PM -0400, Rich Felker wrote:
> On Fri, Aug 10, 2012 at 03:12:55PM -0400, Rich Felker wrote:
> > Other requests?
> 
> Summary so far:
> 
> - significand - easy to do

Actually not quite as easy as I thought. Do we also want the float and
ld versions? And how are they supposed to behave on inf/nan? I think
an optimal implementation would be something like:

if (isnormal(x.d)) return x.rep = (x.rep&mask|exp_0), x.d;
else return scalbn(x, -ilogb(x));

and then just always using scalbnl for ld so as not to introduce more
ld-format-specific code just for legacy purposes.

Anyone up for submitting a patch?

Rich

Re: Priorities for next release?

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2012-08-11 18:25:22 -0400]:
> Actually not quite as easy as I thought. Do we also want the float and
> ld versions? And how are they supposed to behave on inf/nan? I think

bsd only seem to have significand and significandf
so no ld version

actually the same is true for scalb
(musl declares scalbl but only defines
scalb and scalbf, so i guess scalbl can
be reomved from math.h)

> an optimal implementation would be something like:
> 
> if (isnormal(x.d)) return x.rep = (x.rep&mask|exp_0), x.d;
> else return scalbn(x, -ilogb(x));
> 

freebsd just returns scalbn(x, -ilogb(x))
we can do that as well
(eventhough it's not very efficient for normal values)

(the i386 asm version uses fxtract which does the same)

Re: Priorities for next release?

[Luca Barbato]

On 8/11/12 9:56 PM, Rich Felker wrote:
> On Fri, Aug 10, 2012 at 03:12:55PM -0400, Rich Felker wrote:
>> Other requests?
>
> Summary so far:
>
> - significand - easy to do
> - __progname - questionable whether it should be added
> - _XOPEN_SOURCE versions - hard to do, also questionable

I guess for my specific needs I can survive with _BSD_SOURCE probably, 
I'll test again soon.

> - unit tests - very desirable, not going to happen in short time
> - md5/sha crypt - seems harder than i thought; can't find small code

libav ones are based on

http://ubiqx.org/libcifs/source/Auth/MD5.c

and

some bsd/pd code from Steve Reid and Aaron D. Gifford

Hadn't checked where it is but might be a good start.

Re: Priorities for next release?

[Rich Felker]

On Mon, Aug 13, 2012 at 10:40:49AM +0200, Luca Barbato wrote:
> On 8/11/12 9:56 PM, Rich Felker wrote:
> >On Fri, Aug 10, 2012 at 03:12:55PM -0400, Rich Felker wrote:
> >>Other requests?
> >
> >Summary so far:
> >
> >- significand - easy to do
> >- __progname - questionable whether it should be added
> >- _XOPEN_SOURCE versions - hard to do, also questionable
> 
> I guess for my specific needs I can survive with _BSD_SOURCE
> probably, I'll test again soon.

Let me know what specifically it is that you need and I can probably
take care of it.

> >- unit tests - very desirable, not going to happen in short time
> >- md5/sha crypt - seems harder than i thought; can't find small code
> 
> libav ones are based on
> 
> http://ubiqx.org/libcifs/source/Auth/MD5.c

It's LGPL..

Rich

Re: Priorities for next release?

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2012-08-13 09:41:44 -0400]:
> > >- unit tests - very desirable, not going to happen in short time
> > >- md5/sha crypt - seems harder than i thought; can't find small code
> > 
> > libav ones are based on
> > 
> > http://ubiqx.org/libcifs/source/Auth/MD5.c
> 
> It's LGPL..

i have public domain md5 and sha1 based on libtomcrypt

http://port70.net/~nsz/crypt

solar designer has public domain md5

http://openwall.info/wiki/people/solar/software/public-domain-source-code/md5


but i guess for crypt the logic for salt, iterations etc
is the interesting part