[musl] perhaps we should add re[c]allocarray?

[musl] perhaps we should add re[c]allocarray?

[Ariadne Conill]

Hello,

reallocarray and recallocarray are BSD extensions that solve similar issues as 
strlcpy/strlcat, but with array reallocations instead of strings.

reallocarray itself is already part of glibc since 2.28.

Unfortunately, while working on new ifupdown implementation for Alpine, I 
wanted to use recallocarray because it is very helpful in terms of pushing new 
strings to a string array (you will always maintain a NULL-terminated array, 
and you don't have to worry about it) -- but I discovered musl still does not 
have it.

Anyway, I think it would be useful to include both functions in musl 1.2.1.  
If everyone agrees, I'll make a patch.

Ariadne

Re: [musl] perhaps we should add re[c]allocarray?

[Markus Wichmann]

On Tue, Jul 21, 2020 at 04:18:35AM -0600, Ariadne Conill wrote:
> Hello,
>
> reallocarray and recallocarray are BSD extensions that solve similar issues as
> strlcpy/strlcat, but with array reallocations instead of strings.
>
> reallocarray itself is already part of glibc since 2.28.
>
> Unfortunately, while working on new ifupdown implementation for Alpine, I
> wanted to use recallocarray because it is very helpful in terms of pushing new
> strings to a string array (you will always maintain a NULL-terminated array,
> and you don't have to worry about it) -- but I discovered musl still does not
> have it.
>
> Anyway, I think it would be useful to include both functions in musl 1.2.1.
> If everyone agrees, I'll make a patch.
>
> Ariadne
>
>

Seems mostly useless to me. reallocarray() is equivalent to realloc(),
multiplying the last two arguments. And recallocarray() does seem
useful, but moreso as a subroutine. I see little reason to put this into
a standard library.

On a formal point of view, neither of these has been standardized. I can
find an Oracle man page for reallocarray(), but not recallocarray().
Both are OpenBSD extensions. For glibc, I can find reallocarray() (which
mostly wraps realloc()), but no recallocarray() (I checked in the most
recent released version, which is 2.31 as of right now).

It appears, reallocarray() enjoys more widespread adoption than
recallocarray(). Both can, however, be easily found by a compile/link
test. As stated above, however, the necessary functionality can easily
be written in whatever application needs it, so I don't see the point.
I've done that before; it is two lines if you manage your variables
well.

JM2C,
Markus

Re: [musl] perhaps we should add re[c]allocarray?

[Hadrien Lacour]

On Tue, Jul 21, 2020 at 06:56:57PM +0200, Markus Wichmann wrote:
> On Tue, Jul 21, 2020 at 04:18:35AM -0600, Ariadne Conill wrote:
> > Hello,
> >
> > reallocarray and recallocarray are BSD extensions that solve similar issues as
> > strlcpy/strlcat, but with array reallocations instead of strings.
> >
> > reallocarray itself is already part of glibc since 2.28.
> >
> > Unfortunately, while working on new ifupdown implementation for Alpine, I
> > wanted to use recallocarray because it is very helpful in terms of pushing new
> > strings to a string array (you will always maintain a NULL-terminated array,
> > and you don't have to worry about it) -- but I discovered musl still does not
> > have it.
> >
> > Anyway, I think it would be useful to include both functions in musl 1.2.1.
> > If everyone agrees, I'll make a patch.
> >
> > Ariadne
> >
> >
>
> Seems mostly useless to me. reallocarray() is equivalent to realloc(),
> multiplying the last two arguments. And recallocarray() does seem
> useful, but moreso as a subroutine. I see little reason to put this into
> a standard library.
>
> On a formal point of view, neither of these has been standardized. I can
> find an Oracle man page for reallocarray(), but not recallocarray().
> Both are OpenBSD extensions. For glibc, I can find reallocarray() (which
> mostly wraps realloc()), but no recallocarray() (I checked in the most
> recent released version, which is 2.31 as of right now).
>
> It appears, reallocarray() enjoys more widespread adoption than
> recallocarray(). Both can, however, be easily found by a compile/link
> test. As stated above, however, the necessary functionality can easily
> be written in whatever application needs it, so I don't see the point.
> I've done that before; it is two lines if you manage your variables
> well.
>
> JM2C,
> Markus

I'm pretty sure the point of reallocarray is that it checks for overflow during
the multiplication of the arguments.

Re: [musl] perhaps we should add re[c]allocarray?

[Rich Felker]

On Tue, Jul 21, 2020 at 04:18:35AM -0600, Ariadne Conill wrote:
> Hello,
> 
> reallocarray and recallocarray are BSD extensions that solve similar issues as 
> strlcpy/strlcat, but with array reallocations instead of strings.
> 
> reallocarray itself is already part of glibc since 2.28.
> 
> Unfortunately, while working on new ifupdown implementation for Alpine, I 
> wanted to use recallocarray because it is very helpful in terms of pushing new 
> strings to a string array (you will always maintain a NULL-terminated array, 
> and you don't have to worry about it) -- but I discovered musl still does not 
> have it.
> 
> Anyway, I think it would be useful to include both functions in musl 1.2.1.  
> If everyone agrees, I'll make a patch.

reallocarray is a straightforward wrapper around realloc that can be
implemented portably to work with arbitrary underlying malloc and is
fairly non-controversial. I think it was already loosely agreed at
some point that we would eventually support this.

recallocarray presumably needs to zero the new part which means it
needs to know the old exact size, which means it depends on having
either knowledge of implementation internals or a working, exact
malloc_usable_size (AFAIK all legacy/existing ones except musl
mallocng are broken and return a value greater than the originally
allocated size). Implementing it interferes with safety of
overriding/interposing malloc, and therefore I'm fairly strongly
against it unless there's a widepread consensus between implementors
that it should exist.

Is there a strong reason you want recallocarray rather than just
reallocarray?

Rich

Re: [musl] perhaps we should add re[c]allocarray?

[Florian Weimer]

* Rich Felker:

> recallocarray presumably needs to zero the new part which means it
> needs to know the old exact size, which means it depends on having
> either knowledge of implementation internals or a working, exact
> malloc_usable_size (AFAIK all legacy/existing ones except musl
> mallocng are broken and return a value greater than the originally
> allocated size).

The caller has to pass the old member count to recallocarray, in an
additional argument.  I think this avoids this particular issue, and
also makes it easy to achive interposition-safety.

Re: [musl] perhaps we should add re[c]allocarray?

[Leah Neukirchen]

Rich Felker <dalias@libc.org> writes:

> On Tue, Jul 21, 2020 at 04:18:35AM -0600, Ariadne Conill wrote:
>> Hello,
>> 
>> reallocarray and recallocarray are BSD extensions that solve similar issues as 
>> strlcpy/strlcat, but with array reallocations instead of strings.
>> 
>> reallocarray itself is already part of glibc since 2.28.
>> 
>> Unfortunately, while working on new ifupdown implementation for Alpine, I 
>> wanted to use recallocarray because it is very helpful in terms of pushing new 
>> strings to a string array (you will always maintain a NULL-terminated array, 
>> and you don't have to worry about it) -- but I discovered musl still does not 
>> have it.
>> 
>> Anyway, I think it would be useful to include both functions in musl 1.2.1.  
>> If everyone agrees, I'll make a patch.
>
> reallocarray is a straightforward wrapper around realloc that can be
> implemented portably to work with arbitrary underlying malloc and is
> fairly non-controversial. I think it was already loosely agreed at
> some point that we would eventually support this.
>
> recallocarray presumably needs to zero the new part which means it
> needs to know the old exact size, which means it depends on having
> either knowledge of implementation internals or a working, exact
> malloc_usable_size (AFAIK all legacy/existing ones except musl
> mallocng are broken and return a value greater than the originally
> allocated size). Implementing it interferes with safety of
> overriding/interposing malloc, and therefore I'm fairly strongly
> against it unless there's a widepread consensus between implementors
> that it should exist.

No, it's an argument:
void *recallocarray(void *ptr, size_t oldnmemb, size_t newnmemb, size_t size)

-- 
Leah Neukirchen  <leah@vuxu.org>  https://leahneukirchen.org/

Re: [musl] perhaps we should add re[c]allocarray?

[Rich Felker]

On Tue, Jul 21, 2020 at 08:58:04PM +0200, Florian Weimer wrote:
> * Rich Felker:
> 
> > recallocarray presumably needs to zero the new part which means it
> > needs to know the old exact size, which means it depends on having
> > either knowledge of implementation internals or a working, exact
> > malloc_usable_size (AFAIK all legacy/existing ones except musl
> > mallocng are broken and return a value greater than the originally
> > allocated size).
> 
> The caller has to pass the old member count to recallocarray, in an
> additional argument.  I think this avoids this particular issue, and
> also makes it easy to achive interposition-safety.

Ah, great, that makes it a non-issue then, and in that case I have no
significant objections to it.

Rich

Re: [musl] perhaps we should add re[c]allocarray?

[Ariadne Conill]

Hello,

On Tuesday, July 21, 2020 10:56:57 AM MDT Markus Wichmann wrote:
> On Tue, Jul 21, 2020 at 04:18:35AM -0600, Ariadne Conill wrote:
> > Hello,
> > 
> > reallocarray and recallocarray are BSD extensions that solve similar
> > issues as strlcpy/strlcat, but with array reallocations instead of
> > strings.
> > 
> > reallocarray itself is already part of glibc since 2.28.
> > 
> > Unfortunately, while working on new ifupdown implementation for Alpine, I
> > wanted to use recallocarray because it is very helpful in terms of pushing
> > new strings to a string array (you will always maintain a NULL-terminated
> > array, and you don't have to worry about it) -- but I discovered musl
> > still does not have it.
> > 
> > Anyway, I think it would be useful to include both functions in musl
> > 1.2.1.
> > If everyone agrees, I'll make a patch.
> > 
> > Ariadne
> 
> Seems mostly useless to me. reallocarray() is equivalent to realloc(),
> multiplying the last two arguments. And recallocarray() does seem
> useful, but moreso as a subroutine. I see little reason to put this into
> a standard library.

The reason is that we would like to see people use these routines instead of 
fussing with realloc() directly because they do the right thing.  It is better 
to provide the right thing in the standard library instead of having people 
mess it up with their own implementation.

> On a formal point of view, neither of these has been standardized. I can
> find an Oracle man page for reallocarray(), but not recallocarray().
> Both are OpenBSD extensions. For glibc, I can find reallocarray() (which
> mostly wraps realloc()), but no recallocarray() (I checked in the most
> recent released version, which is 2.31 as of right now).

As I previously stated, both are BSD extensions, so I do not understand why 
you are mentioning it again.  At any rate, I plan to propose these extensions 
for inclusion in next POSIX revision.  Just haven't gotten around to writing 
to the Austin Group yet.

> It appears, reallocarray() enjoys more widespread adoption than
> recallocarray(). Both can, however, be easily found by a compile/link
> test. As stated above, however, the necessary functionality can easily
> be written in whatever application needs it, so I don't see the point.
> I've done that before; it is two lines if you manage your variables
> well.

While it is possible to probe for these functions using autoconf or meson or 
whatever, Alpine approaches these concerns from the standpoint that the libc 
provides what Alpine requires for its own utilities.  For now, we will carry 
our own recallocarray in ifupdown, but it would be nice to drop this at some 
point.  That is what *this* thread is about.

Ariadne

Re: [musl] perhaps we should add re[c]allocarray?

[Ariadne Conill]

Hello,

On Tuesday, July 21, 2020 2:40:53 PM MDT you wrote:
> On Tue, Jul 21, 2020 at 08:58:04PM +0200, Florian Weimer wrote:
> > * Rich Felker:
> > > recallocarray presumably needs to zero the new part which means it
> > > needs to know the old exact size, which means it depends on having
> > > either knowledge of implementation internals or a working, exact
> > > malloc_usable_size (AFAIK all legacy/existing ones except musl
> > > mallocng are broken and return a value greater than the originally
> > > allocated size).
> > 
> > The caller has to pass the old member count to recallocarray, in an
> > additional argument.  I think this avoids this particular issue, and
> > also makes it easy to achive interposition-safety.
> 
> Ah, great, that makes it a non-issue then, and in that case I have no
> significant objections to it.

Okay great.  I will work on reallocarray() first and then follow up with 
recallocarray().

Ariadne

Re: [musl] perhaps we should add re[c]allocarray?

[Rich Felker]

On Tue, Jul 21, 2020 at 05:21:03PM -0600, Ariadne Conill wrote:
> Hello,
> 
> On Tuesday, July 21, 2020 2:40:53 PM MDT you wrote:
> > On Tue, Jul 21, 2020 at 08:58:04PM +0200, Florian Weimer wrote:
> > > * Rich Felker:
> > > > recallocarray presumably needs to zero the new part which means it
> > > > needs to know the old exact size, which means it depends on having
> > > > either knowledge of implementation internals or a working, exact
> > > > malloc_usable_size (AFAIK all legacy/existing ones except musl
> > > > mallocng are broken and return a value greater than the originally
> > > > allocated size).
> > > 
> > > The caller has to pass the old member count to recallocarray, in an
> > > additional argument.  I think this avoids this particular issue, and
> > > also makes it easy to achive interposition-safety.
> > 
> > Ah, great, that makes it a non-issue then, and in that case I have no
> > significant objections to it.
> 
> Okay great.  I will work on reallocarray() first and then follow up with 
> recallocarray().

Yes, reallocarray should be very straightforward. I think
recallocarray should probably involve a slight refactor of calloc.c to
make mal0_clear external so it can be reused; otherwise recallocarray
would be significantly worse than calloc/memcpy/free since it would
fault in all pages right away.

Rich

Re: [musl] perhaps we should add re[c]allocarray?

[Ariadne Conill]

Hello,

On Tuesday, July 21, 2020 6:21:16 PM MDT Rich Felker wrote:
> On Tue, Jul 21, 2020 at 05:21:03PM -0600, Ariadne Conill wrote:
> > Hello,
> > 
> > On Tuesday, July 21, 2020 2:40:53 PM MDT you wrote:
> > > On Tue, Jul 21, 2020 at 08:58:04PM +0200, Florian Weimer wrote:
> > > > * Rich Felker:
> > > > > recallocarray presumably needs to zero the new part which means it
> > > > > needs to know the old exact size, which means it depends on having
> > > > > either knowledge of implementation internals or a working, exact
> > > > > malloc_usable_size (AFAIK all legacy/existing ones except musl
> > > > > mallocng are broken and return a value greater than the originally
> > > > > allocated size).
> > > > 
> > > > The caller has to pass the old member count to recallocarray, in an
> > > > additional argument.  I think this avoids this particular issue, and
> > > > also makes it easy to achive interposition-safety.
> > > 
> > > Ah, great, that makes it a non-issue then, and in that case I have no
> > > significant objections to it.
> > 
> > Okay great.  I will work on reallocarray() first and then follow up with
> > recallocarray().
> 
> Yes, reallocarray should be very straightforward. I think
> recallocarray should probably involve a slight refactor of calloc.c to
> make mal0_clear external so it can be reused; otherwise recallocarray
> would be significantly worse than calloc/memcpy/free since it would
> fault in all pages right away.

Yeah, I was going to alias mal0_clear to __malloc_mal0_clear or similar as 
part of recallocarray work.

Ariadne