SoX & pipe rewinding

SoX & pipe rewinding

[ojab]

[-- Attachment #1: Type: text/plain, Size: 207 bytes --]

Hi list,
SoX use some kind of hackery for pipe rewinding (see the attached file). 
So I have two questions:
1. If something like that possible using musl?
2. Is there any musl specific #define?

//wbr ojab


[-- Attachment #2: rewind_pipe.txt --]
[-- Type: text/plain, Size: 854 bytes --]


/* Hack to rewind pipes (a small amount).
 * Works by resetting the FILE buffer pointer */
static void UNUSED rewind_pipe(FILE * fp)
{
/* _FSTDIO is for Torek stdio (i.e. most BSD-derived libc's)
 * In theory, we no longer need to check _NEWLIB_VERSION or __APPLE__ */
#if defined _FSTDIO || defined _NEWLIB_VERSION || defined __APPLE__
  fp->_p -= AUTO_DETECT_SIZE;
  fp->_r += AUTO_DETECT_SIZE;
#elif defined __GLIBC__
  fp->_IO_read_ptr = fp->_IO_read_base;
#elif defined _MSC_VER || defined __MINGW_H || defined _ISO_STDIO_ISO_H
  fp->_ptr = fp->_base;
#else
  /* To fix this #error, either simply remove the #error line and live without
   * file-type detection with pipes, or add support for your compiler in the
   * lines above.  Test with cat monkey.au | ./sox --info - */
  #error FIX NEEDED HERE
  #define NO_REWIND_PIPE
  (void)fp;
#endif
}

Re: SoX & pipe rewinding

[Rich Felker]

On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote:
> Hi list,
> SoX use some kind of hackery for pipe rewinding (see the attached
> file). So I have two questions:
> 1. If something like that possible using musl?
> 2. Is there any musl specific #define?

I don't believe so. Not only does this hack depend on knowing the
internals of FILE (which are intentionally opaque on musl); it also
depends on the assumption that the data is still present in the buffer
after it's consumed, which is invalid in general.

No idea why programmers insist on doing stuff like this rather than
fixing their design issues...

Rich

Re: SoX & pipe rewinding

[ojab]

On 14.12.2012 17:40, Rich Felker wrote:
> On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote:
>> >Hi list,
>> >SoX use some kind of hackery for pipe rewinding (see the attached
>> >file). So I have two questions:
>> >1. If something like that possible using musl?
>> >2. Is there any musl specific #define?
> I don't believe so. Not only does this hack depend on knowing the
> internals of FILE (which are intentionally opaque on musl); it also
> depends on the assumption that the data is still present in the buffer
> after it's consumed, which is invalid in general.
>
> No idea why programmers insist on doing stuff like this rather than
> fixing their design issues...
>
> Rich
>

It's completely optional (check the #else part), so question is (2): How 
we can check for musl and disable pipe rewinding in this case.
If there is no musl-specific define, and there is intention not to add 
one — I'll file a bug in SoX tracker with request to remove an #error.

//wbr ojab.

Re: SoX & pipe rewinding

[John Spencer]

On 12/14/2012 02:51 PM, ojab wrote:
>
>
> It's completely optional (check the #else part), so question is (2): 
> How we can check for musl and disable pipe rewinding in this case.
> If there is no musl-specific define, and there is intention not to add 
> one — I'll file a bug in SoX tracker with request to remove an #error.
>
> //wbr ojab.


i think this function is completely unused as the UNUSED keyword in the 
declaration suggests.
in that case, simply removing the function would be sufficient...

Re: SoX & pipe rewinding

[ojab]

On 14.12.2012 20:47, John Spencer wrote:
> On 12/14/2012 02:51 PM, ojab wrote:
>
>
> i think this function is completely unused as the UNUSED keyword in the
> declaration suggests.
> in that case, simply removing the function would be sufficient...
>

Unfortunately it is used, but parts of code where it's used are guarded 
by "#ifndef NO_REWIND_PIPE".
So in the "#else" case it is unused, most likely keyword was added 
because of that.

//wbr ojab

Re: SoX & pipe rewinding

[Rich Felker]

On Fri, Dec 14, 2012 at 05:51:40PM +0400, ojab wrote:
> On 14.12.2012 17:40, Rich Felker wrote:
> >On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote:
> >>>Hi list,
> >>>SoX use some kind of hackery for pipe rewinding (see the attached
> >>>file). So I have two questions:
> >>>1. If something like that possible using musl?
> >>>2. Is there any musl specific #define?
> >I don't believe so. Not only does this hack depend on knowing the
> >internals of FILE (which are intentionally opaque on musl); it also
> >depends on the assumption that the data is still present in the buffer
> >after it's consumed, which is invalid in general.
> >
> >No idea why programmers insist on doing stuff like this rather than
> >fixing their design issues...
> >
> >Rich
> >
> 
> It's completely optional (check the #else part), so question is (2):
> How we can check for musl and disable pipe rewinding in this case.
> If there is no musl-specific define, and there is intention not to
> add one — I'll file a bug in SoX tracker with request to remove an
> #error.

There's intentionally no musl-specific #define. Some people have
complained about that, but if there had been one, the state of musl
compatibility would be a ton worse right now, because lots of software
would be crippling itself by using #ifdef __musl__ or similar to
disable features that musl 0.6.x didn't have...

The right solution to using optional features is to test for their
existence, but in this case, sox is not using an extension feature but
poking at the internals of some known implementations and throwing
#error for everything else. This is just really bad behavior. The
default case should be portable, i.e. should just silently omit the
non-portable pipe rewinding code.

Rich

Re: SoX & pipe rewinding

[ojab]

On 14.12.2012 23:09, Rich Felker wrote:
> On Fri, Dec 14, 2012 at 05:51:40PM +0400, ojab wrote:
>
> There's intentionally no musl-specific #define. Some people have
> complained about that, but if there had been one, the state of musl
> compatibility would be a ton worse right now, because lots of software
> would be crippling itself by using #ifdef __musl__ or similar to
> disable features that musl 0.6.x didn't have...
>
> The right solution to using optional features is to test for their
> existence, but in this case, sox is not using an extension feature but
> poking at the internals of some known implementations and throwing
> #error for everything else. This is just really bad behavior. The
> default case should be portable, i.e. should just silently omit the
> non-portable pipe rewinding code.
>
> Rich
>

Filled 
https://sourceforge.net/tracker/?func=detail&aid=3596097&group_id=10706&atid=110706

//wbr ojab

Re: SoX & pipe rewinding

[Rich Felker]

On Sat, Dec 15, 2012 at 01:11:22AM +0400, ojab wrote:
> On 14.12.2012 23:09, Rich Felker wrote:
> >On Fri, Dec 14, 2012 at 05:51:40PM +0400, ojab wrote:
> >
> >There's intentionally no musl-specific #define. Some people have
> >complained about that, but if there had been one, the state of musl
> >compatibility would be a ton worse right now, because lots of software
> >would be crippling itself by using #ifdef __musl__ or similar to
> >disable features that musl 0.6.x didn't have...
> >
> >The right solution to using optional features is to test for their
> >existence, but in this case, sox is not using an extension feature but
> >poking at the internals of some known implementations and throwing
> >#error for everything else. This is just really bad behavior. The
> >default case should be portable, i.e. should just silently omit the
> >non-portable pipe rewinding code.
> 
> Filled https://sourceforge.net/tracker/?func=detail&aid=3596097&group_id=10706&atid=110706

Thanks. It might be nice to also add a link to this thread, which
makes it clear that our position is that the layout of the FILE
structure and the buffering behavior of musl's stdio are both
considered opaque to applications; hacks to detect musl and poke at
the current version of the internals, which possible, would not
necessarily be compatible with future versions of musl, i.e. it would
result in a binary that might break/crash horribly when libc.so is
upgraded.

BTW, there is one 100% safe way to _attempt_ rewinding a pipe:
repeatedly call ungetc() for each character you read. The C standard
does not guarantee more than 1 byte of pushback, but most
implementations allow more under most circumstances, and the C
standard does require that the implementation report the success or
failure of ungetc() to push back a byte. This approach would always
work with the current stdio implementation in musl, and if any
internal changes prevented it from working in the future, the
application would at least see the failure (in the form of a failure
return from ungetc()) and would be able to react appropriately rather
than crashing due to corrupted internal stdio state. However, to use
this approach, I think the sox rewind-pipe function would have to be
improved to take a pointer to the header block that was read for
probing purposes, so that it could push-back these bytes one at a
time.

Rich

Re: SoX & pipe rewinding

[Rob Landley]

On 12/14/2012 07:40:09 AM, Rich Felker wrote:
> On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote:
> > Hi list,
> > SoX use some kind of hackery for pipe rewinding (see the attached
> > file). So I have two questions:
> > 1. If something like that possible using musl?

Create a wrapper FILE object that passes through most operations but  
copies incoming data to a ring buffer you can re-read previous data out  
of.

> > 2. Is there any musl specific #define?
> 
> I don't believe so.

Which is annoying.

Imagine an OS kernel that refused to identify itself because its author  
didn't want software knowing it was running on it. Because it's PERFECT  
and knows it's perfect sight unseen, and thus nobody out in the world  
should ever need to work around its perfection, and anyone who uses it  
is _required_ not to.

> No idea why programmers insist on doing stuff like this rather than
> fixing their design issues...

It's such a crazy thing to want to do that both posix and ISO C  
standardize an ungetc() function.

Other programmers don't write code the way you would write it. And when  
you try to force them, you make the result even uglier.

For example, if a program really wants to know whether or not it's  
running against musl right now, a dynamically linked program can do  
something like "strings /proc/self/exe | grep ld-musl", and just refuse  
to work staticaly linked. (Yes, this is utterly horrible, but if you  
refuse to have an #define people can check I just about guarantee you  
somebody will do this sooner or later. Or worse.)

Rob

Re: SoX & pipe rewinding

[Rich Felker]

On Fri, Dec 14, 2012 at 04:17:34PM -0600, Rob Landley wrote:
> On 12/14/2012 07:40:09 AM, Rich Felker wrote:
> >On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote:
> >> Hi list,
> >> SoX use some kind of hackery for pipe rewinding (see the attached
> >> file). So I have two questions:
> >> 1. If something like that possible using musl?
> 
> Create a wrapper FILE object that passes through most operations but
> copies incoming data to a ring buffer you can re-read previous data
> out of.

You can create your own IO buffering layer in place of stdio, yes. Or
you can use 2 FILE objects stacked with a pipe and threads between
them. Or you can use a temp FILE and swap back to the real one once
the temp one runs out of data.

> >> 2. Is there any musl specific #define?
> >
> >I don't believe so.
> 
> Which is annoying.
> 
> Imagine an OS kernel that refused to identify itself because its
> author didn't want software knowing it was running on it. Because
> it's PERFECT and knows it's perfect sight unseen, and thus nobody
> out in the world should ever need to work around its perfection, and
> anyone who uses it is _required_ not to.

It's not because it's perfect. It's because the only valid way to know
whether a particular extension or bug exists is to TEST for it. Note
that applications and modules which tweaked their behavior based on
kernel version numbers are also horribly broken because some distros
have kernels with fixes and features backported.

The idiom of:

    if (version == X)
        assume_feature_i_once_observed_on_version_X;

is just fundamentally broken. Testing is the only valid approach to
answering the question of whether a feature is present -- either
testing compiling/linking to it, or testing for a macro that indicates
the presence of a specific feature (like the _POSIX_THREADS macro
does) rather than a whole library version.

> >No idea why programmers insist on doing stuff like this rather than
> >fixing their design issues...
> 
> It's such a crazy thing to want to do that both posix and ISO C
> standardize an ungetc() function.

If you see my followup email, I explained how the effect could be
achieved, on some systems under some conditions, via ungetc.

> Other programmers don't write code the way you would write it. And
> when you try to force them, you make the result even uglier.

I don't try to force them. All I do is ensure that musl remains
unconstrained to innovate by improving the underlying implementation
of things, unlike glibc which is tied down to all the legacy
implementation choices they made.

> For example, if a program really wants to know whether or not it's
> running against musl right now, a dynamically linked program can do
> something like "strings /proc/self/exe | grep ld-musl", and just
> refuse to work staticaly linked. (Yes, this is utterly horrible, but
> if you refuse to have an #define people can check I just about
> guarantee you somebody will do this sooner or later. Or worse.)

Of course apps can do this. And their authors will be the ones who
look like fools in the court of public opinion. I have explained
thoroughly the reasoning behind these issues, and every time it's come
up, my position has been demonstrated _empirically_ right as well
(i.e. the thing somebody was trying to do using #ifdef __musl__ was
obsolete within a month or two and would have _broken_ building with
newer versions). Many times it's also just led to musl adopting
extension interfaces that were not difficult to add and which served
to avoid ugly hacks (e.g. stdio/ext2.c).

Actually this functionality (rewinding buffer) is something I feel
similar about. I would have no big objection to providing a "rewind in
buffer, reporting success for failure, subject to the limitation that
whether or not the data is still there is unspecified" function in
musl, which applications could test for and use, if there were a big
demand for this functionality. What is not acceptable is locking in
the layout of the FILE structure any more than it's already
constrained, or locking in the constraint that the implementation
can't discard buffer contents that have already been read.

Rich