* SoX & pipe rewinding @ 2012-12-14 13:35 ojab 2012-12-14 13:40 ` Rich Felker 0 siblings, 1 reply; 10+ messages in thread From: ojab @ 2012-12-14 13:35 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 207 bytes --] Hi list, SoX use some kind of hackery for pipe rewinding (see the attached file). So I have two questions: 1. If something like that possible using musl? 2. Is there any musl specific #define? //wbr ojab [-- Attachment #2: rewind_pipe.txt --] [-- Type: text/plain, Size: 854 bytes --] /* Hack to rewind pipes (a small amount). * Works by resetting the FILE buffer pointer */ static void UNUSED rewind_pipe(FILE * fp) { /* _FSTDIO is for Torek stdio (i.e. most BSD-derived libc's) * In theory, we no longer need to check _NEWLIB_VERSION or __APPLE__ */ #if defined _FSTDIO || defined _NEWLIB_VERSION || defined __APPLE__ fp->_p -= AUTO_DETECT_SIZE; fp->_r += AUTO_DETECT_SIZE; #elif defined __GLIBC__ fp->_IO_read_ptr = fp->_IO_read_base; #elif defined _MSC_VER || defined __MINGW_H || defined _ISO_STDIO_ISO_H fp->_ptr = fp->_base; #else /* To fix this #error, either simply remove the #error line and live without * file-type detection with pipes, or add support for your compiler in the * lines above. Test with cat monkey.au | ./sox --info - */ #error FIX NEEDED HERE #define NO_REWIND_PIPE (void)fp; #endif } ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: SoX & pipe rewinding 2012-12-14 13:35 SoX & pipe rewinding ojab @ 2012-12-14 13:40 ` Rich Felker 2012-12-14 13:51 ` ojab 2012-12-14 22:17 ` Rob Landley 0 siblings, 2 replies; 10+ messages in thread From: Rich Felker @ 2012-12-14 13:40 UTC (permalink / raw) To: musl On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote: > Hi list, > SoX use some kind of hackery for pipe rewinding (see the attached > file). So I have two questions: > 1. If something like that possible using musl? > 2. Is there any musl specific #define? I don't believe so. Not only does this hack depend on knowing the internals of FILE (which are intentionally opaque on musl); it also depends on the assumption that the data is still present in the buffer after it's consumed, which is invalid in general. No idea why programmers insist on doing stuff like this rather than fixing their design issues... Rich ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: SoX & pipe rewinding 2012-12-14 13:40 ` Rich Felker @ 2012-12-14 13:51 ` ojab 2012-12-14 16:47 ` John Spencer 2012-12-14 19:09 ` Rich Felker 2012-12-14 22:17 ` Rob Landley 1 sibling, 2 replies; 10+ messages in thread From: ojab @ 2012-12-14 13:51 UTC (permalink / raw) To: musl On 14.12.2012 17:40, Rich Felker wrote: > On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote: >> >Hi list, >> >SoX use some kind of hackery for pipe rewinding (see the attached >> >file). So I have two questions: >> >1. If something like that possible using musl? >> >2. Is there any musl specific #define? > I don't believe so. Not only does this hack depend on knowing the > internals of FILE (which are intentionally opaque on musl); it also > depends on the assumption that the data is still present in the buffer > after it's consumed, which is invalid in general. > > No idea why programmers insist on doing stuff like this rather than > fixing their design issues... > > Rich > It's completely optional (check the #else part), so question is (2): How we can check for musl and disable pipe rewinding in this case. If there is no musl-specific define, and there is intention not to add one — I'll file a bug in SoX tracker with request to remove an #error. //wbr ojab. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: SoX & pipe rewinding 2012-12-14 13:51 ` ojab @ 2012-12-14 16:47 ` John Spencer 2012-12-14 17:57 ` ojab 2012-12-14 19:09 ` Rich Felker 1 sibling, 1 reply; 10+ messages in thread From: John Spencer @ 2012-12-14 16:47 UTC (permalink / raw) To: musl On 12/14/2012 02:51 PM, ojab wrote: > > > It's completely optional (check the #else part), so question is (2): > How we can check for musl and disable pipe rewinding in this case. > If there is no musl-specific define, and there is intention not to add > one — I'll file a bug in SoX tracker with request to remove an #error. > > //wbr ojab. i think this function is completely unused as the UNUSED keyword in the declaration suggests. in that case, simply removing the function would be sufficient... ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: SoX & pipe rewinding 2012-12-14 16:47 ` John Spencer @ 2012-12-14 17:57 ` ojab 0 siblings, 0 replies; 10+ messages in thread From: ojab @ 2012-12-14 17:57 UTC (permalink / raw) To: musl On 14.12.2012 20:47, John Spencer wrote: > On 12/14/2012 02:51 PM, ojab wrote: > > > i think this function is completely unused as the UNUSED keyword in the > declaration suggests. > in that case, simply removing the function would be sufficient... > Unfortunately it is used, but parts of code where it's used are guarded by "#ifndef NO_REWIND_PIPE". So in the "#else" case it is unused, most likely keyword was added because of that. //wbr ojab ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: SoX & pipe rewinding 2012-12-14 13:51 ` ojab 2012-12-14 16:47 ` John Spencer @ 2012-12-14 19:09 ` Rich Felker 2012-12-14 21:11 ` ojab 1 sibling, 1 reply; 10+ messages in thread From: Rich Felker @ 2012-12-14 19:09 UTC (permalink / raw) To: musl On Fri, Dec 14, 2012 at 05:51:40PM +0400, ojab wrote: > On 14.12.2012 17:40, Rich Felker wrote: > >On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote: > >>>Hi list, > >>>SoX use some kind of hackery for pipe rewinding (see the attached > >>>file). So I have two questions: > >>>1. If something like that possible using musl? > >>>2. Is there any musl specific #define? > >I don't believe so. Not only does this hack depend on knowing the > >internals of FILE (which are intentionally opaque on musl); it also > >depends on the assumption that the data is still present in the buffer > >after it's consumed, which is invalid in general. > > > >No idea why programmers insist on doing stuff like this rather than > >fixing their design issues... > > > >Rich > > > > It's completely optional (check the #else part), so question is (2): > How we can check for musl and disable pipe rewinding in this case. > If there is no musl-specific define, and there is intention not to > add one — I'll file a bug in SoX tracker with request to remove an > #error. There's intentionally no musl-specific #define. Some people have complained about that, but if there had been one, the state of musl compatibility would be a ton worse right now, because lots of software would be crippling itself by using #ifdef __musl__ or similar to disable features that musl 0.6.x didn't have... The right solution to using optional features is to test for their existence, but in this case, sox is not using an extension feature but poking at the internals of some known implementations and throwing #error for everything else. This is just really bad behavior. The default case should be portable, i.e. should just silently omit the non-portable pipe rewinding code. Rich ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: SoX & pipe rewinding 2012-12-14 19:09 ` Rich Felker @ 2012-12-14 21:11 ` ojab 2012-12-14 21:21 ` Rich Felker 0 siblings, 1 reply; 10+ messages in thread From: ojab @ 2012-12-14 21:11 UTC (permalink / raw) To: musl On 14.12.2012 23:09, Rich Felker wrote: > On Fri, Dec 14, 2012 at 05:51:40PM +0400, ojab wrote: > > There's intentionally no musl-specific #define. Some people have > complained about that, but if there had been one, the state of musl > compatibility would be a ton worse right now, because lots of software > would be crippling itself by using #ifdef __musl__ or similar to > disable features that musl 0.6.x didn't have... > > The right solution to using optional features is to test for their > existence, but in this case, sox is not using an extension feature but > poking at the internals of some known implementations and throwing > #error for everything else. This is just really bad behavior. The > default case should be portable, i.e. should just silently omit the > non-portable pipe rewinding code. > > Rich > Filled https://sourceforge.net/tracker/?func=detail&aid=3596097&group_id=10706&atid=110706 //wbr ojab ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: SoX & pipe rewinding 2012-12-14 21:11 ` ojab @ 2012-12-14 21:21 ` Rich Felker 0 siblings, 0 replies; 10+ messages in thread From: Rich Felker @ 2012-12-14 21:21 UTC (permalink / raw) To: musl On Sat, Dec 15, 2012 at 01:11:22AM +0400, ojab wrote: > On 14.12.2012 23:09, Rich Felker wrote: > >On Fri, Dec 14, 2012 at 05:51:40PM +0400, ojab wrote: > > > >There's intentionally no musl-specific #define. Some people have > >complained about that, but if there had been one, the state of musl > >compatibility would be a ton worse right now, because lots of software > >would be crippling itself by using #ifdef __musl__ or similar to > >disable features that musl 0.6.x didn't have... > > > >The right solution to using optional features is to test for their > >existence, but in this case, sox is not using an extension feature but > >poking at the internals of some known implementations and throwing > >#error for everything else. This is just really bad behavior. The > >default case should be portable, i.e. should just silently omit the > >non-portable pipe rewinding code. > > Filled https://sourceforge.net/tracker/?func=detail&aid=3596097&group_id=10706&atid=110706 Thanks. It might be nice to also add a link to this thread, which makes it clear that our position is that the layout of the FILE structure and the buffering behavior of musl's stdio are both considered opaque to applications; hacks to detect musl and poke at the current version of the internals, which possible, would not necessarily be compatible with future versions of musl, i.e. it would result in a binary that might break/crash horribly when libc.so is upgraded. BTW, there is one 100% safe way to _attempt_ rewinding a pipe: repeatedly call ungetc() for each character you read. The C standard does not guarantee more than 1 byte of pushback, but most implementations allow more under most circumstances, and the C standard does require that the implementation report the success or failure of ungetc() to push back a byte. This approach would always work with the current stdio implementation in musl, and if any internal changes prevented it from working in the future, the application would at least see the failure (in the form of a failure return from ungetc()) and would be able to react appropriately rather than crashing due to corrupted internal stdio state. However, to use this approach, I think the sox rewind-pipe function would have to be improved to take a pointer to the header block that was read for probing purposes, so that it could push-back these bytes one at a time. Rich ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: SoX & pipe rewinding 2012-12-14 13:40 ` Rich Felker 2012-12-14 13:51 ` ojab @ 2012-12-14 22:17 ` Rob Landley 2012-12-15 0:05 ` Rich Felker 1 sibling, 1 reply; 10+ messages in thread From: Rob Landley @ 2012-12-14 22:17 UTC (permalink / raw) To: musl; +Cc: musl On 12/14/2012 07:40:09 AM, Rich Felker wrote: > On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote: > > Hi list, > > SoX use some kind of hackery for pipe rewinding (see the attached > > file). So I have two questions: > > 1. If something like that possible using musl? Create a wrapper FILE object that passes through most operations but copies incoming data to a ring buffer you can re-read previous data out of. > > 2. Is there any musl specific #define? > > I don't believe so. Which is annoying. Imagine an OS kernel that refused to identify itself because its author didn't want software knowing it was running on it. Because it's PERFECT and knows it's perfect sight unseen, and thus nobody out in the world should ever need to work around its perfection, and anyone who uses it is _required_ not to. > No idea why programmers insist on doing stuff like this rather than > fixing their design issues... It's such a crazy thing to want to do that both posix and ISO C standardize an ungetc() function. Other programmers don't write code the way you would write it. And when you try to force them, you make the result even uglier. For example, if a program really wants to know whether or not it's running against musl right now, a dynamically linked program can do something like "strings /proc/self/exe | grep ld-musl", and just refuse to work staticaly linked. (Yes, this is utterly horrible, but if you refuse to have an #define people can check I just about guarantee you somebody will do this sooner or later. Or worse.) Rob ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: SoX & pipe rewinding 2012-12-14 22:17 ` Rob Landley @ 2012-12-15 0:05 ` Rich Felker 0 siblings, 0 replies; 10+ messages in thread From: Rich Felker @ 2012-12-15 0:05 UTC (permalink / raw) To: musl On Fri, Dec 14, 2012 at 04:17:34PM -0600, Rob Landley wrote: > On 12/14/2012 07:40:09 AM, Rich Felker wrote: > >On Fri, Dec 14, 2012 at 05:35:52PM +0400, ojab wrote: > >> Hi list, > >> SoX use some kind of hackery for pipe rewinding (see the attached > >> file). So I have two questions: > >> 1. If something like that possible using musl? > > Create a wrapper FILE object that passes through most operations but > copies incoming data to a ring buffer you can re-read previous data > out of. You can create your own IO buffering layer in place of stdio, yes. Or you can use 2 FILE objects stacked with a pipe and threads between them. Or you can use a temp FILE and swap back to the real one once the temp one runs out of data. > >> 2. Is there any musl specific #define? > > > >I don't believe so. > > Which is annoying. > > Imagine an OS kernel that refused to identify itself because its > author didn't want software knowing it was running on it. Because > it's PERFECT and knows it's perfect sight unseen, and thus nobody > out in the world should ever need to work around its perfection, and > anyone who uses it is _required_ not to. It's not because it's perfect. It's because the only valid way to know whether a particular extension or bug exists is to TEST for it. Note that applications and modules which tweaked their behavior based on kernel version numbers are also horribly broken because some distros have kernels with fixes and features backported. The idiom of: if (version == X) assume_feature_i_once_observed_on_version_X; is just fundamentally broken. Testing is the only valid approach to answering the question of whether a feature is present -- either testing compiling/linking to it, or testing for a macro that indicates the presence of a specific feature (like the _POSIX_THREADS macro does) rather than a whole library version. > >No idea why programmers insist on doing stuff like this rather than > >fixing their design issues... > > It's such a crazy thing to want to do that both posix and ISO C > standardize an ungetc() function. If you see my followup email, I explained how the effect could be achieved, on some systems under some conditions, via ungetc. > Other programmers don't write code the way you would write it. And > when you try to force them, you make the result even uglier. I don't try to force them. All I do is ensure that musl remains unconstrained to innovate by improving the underlying implementation of things, unlike glibc which is tied down to all the legacy implementation choices they made. > For example, if a program really wants to know whether or not it's > running against musl right now, a dynamically linked program can do > something like "strings /proc/self/exe | grep ld-musl", and just > refuse to work staticaly linked. (Yes, this is utterly horrible, but > if you refuse to have an #define people can check I just about > guarantee you somebody will do this sooner or later. Or worse.) Of course apps can do this. And their authors will be the ones who look like fools in the court of public opinion. I have explained thoroughly the reasoning behind these issues, and every time it's come up, my position has been demonstrated _empirically_ right as well (i.e. the thing somebody was trying to do using #ifdef __musl__ was obsolete within a month or two and would have _broken_ building with newer versions). Many times it's also just led to musl adopting extension interfaces that were not difficult to add and which served to avoid ugly hacks (e.g. stdio/ext2.c). Actually this functionality (rewinding buffer) is something I feel similar about. I would have no big objection to providing a "rewind in buffer, reporting success for failure, subject to the limitation that whether or not the data is still there is unspecified" function in musl, which applications could test for and use, if there were a big demand for this functionality. What is not acceptable is locking in the layout of the FILE structure any more than it's already constrained, or locking in the constraint that the implementation can't discard buffer contents that have already been read. Rich ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2012-12-15 0:05 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-12-14 13:35 SoX & pipe rewinding ojab 2012-12-14 13:40 ` Rich Felker 2012-12-14 13:51 ` ojab 2012-12-14 16:47 ` John Spencer 2012-12-14 17:57 ` ojab 2012-12-14 19:09 ` Rich Felker 2012-12-14 21:11 ` ojab 2012-12-14 21:21 ` Rich Felker 2012-12-14 22:17 ` Rob Landley 2012-12-15 0:05 ` Rich Felker
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/musl/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).