* freebsd problems with carriage return
@ 2009-03-24 10:14 Atom Smasher
2009-03-24 11:47 ` Stephane Chazelas
0 siblings, 1 reply; 8+ messages in thread
From: Atom Smasher @ 2009-03-24 10:14 UTC (permalink / raw)
To: zsh-users
on linux this works as expected:
rsync --progress /from /to | while read -d "`print '\r'`" n
do
print " foo: $n"
done
as rsync uses a carriage return to update the progress of the file
transfer, the "read" builtin uses '\r' as a delimiter, and prints the
output as expected, one line at a time.
on freebsd it doesn't work at all. i suspect the bug may be in bsd's
libraries, because a similar thing happens on linux and fails to happen on
freebsd with:
rsync --progress /from /to | tr '\r' '\n'
of course linux is using gnu-tr and freebsd is using bsd-tr.
can anyone help identify the source of the problem, and/or a workaround?
thanks...!
--
...atom
________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------
"War will cease when men refuse to fight."
-- F. Hansen
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: freebsd problems with carriage return
2009-03-24 10:14 freebsd problems with carriage return Atom Smasher
@ 2009-03-24 11:47 ` Stephane Chazelas
2009-03-24 21:36 ` Atom Smasher
0 siblings, 1 reply; 8+ messages in thread
From: Stephane Chazelas @ 2009-03-24 11:47 UTC (permalink / raw)
To: Atom Smasher; +Cc: zsh-users
2009-03-24 23:14:09 +1300, Atom Smasher:
> on linux this works as expected:
>
> rsync --progress /from /to | while read -d "`print '\r'`" n
See $'\r' instead of "`print '\r'`"
> do
> print " foo: $n"
should be print -r, print without -r expands the \ sequences.
You should probably use IFS= read -r instead of read as well.
> done
>
> as rsync uses a carriage return to update the progress of the file
> transfer, the "read" builtin uses '\r' as a delimiter, and prints the
> output as expected, one line at a time.
>
> on freebsd it doesn't work at all.
In which way does it not work?
> i suspect the bug may be in bsd's
> libraries, because a similar thing happens on linux and fails to happen on
> freebsd with:
> rsync --progress /from /to | tr '\r' '\n'
>
> of course linux is using gnu-tr and freebsd is using bsd-tr.
>
> can anyone help identify the source of the problem, and/or a workaround?
[...]
run
rsync --progress /from /to | sed -n l
(or od -c)
to see what rsync actually outputs.
Also, in case rsync outputs it on stderr, try:
rsync --progress /from /to 2>&1 | sed -n l
--
Stephane
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: freebsd problems with carriage return
2009-03-24 11:47 ` Stephane Chazelas
@ 2009-03-24 21:36 ` Atom Smasher
2009-03-25 8:18 ` Benjamin R. Haskell
2009-03-25 11:02 ` Stephane Chazelas
0 siblings, 2 replies; 8+ messages in thread
From: Atom Smasher @ 2009-03-24 21:36 UTC (permalink / raw)
To: Stephane Chazelas; +Cc: zsh-users
On Tue, 24 Mar 2009, Stephane Chazelas wrote:
> 2009-03-24 23:14:09 +1300, Atom Smasher:
>> on linux this works as expected:
>>
>> rsync --progress /from /to | while read -d "`print '\r'`" n
>
> See $'\r' instead of "`print '\r'`"
>
>> do
>> print " foo: $n"
>
> should be print -r, print without -r expands the \ sequences.
>
> You should probably use IFS= read -r instead of read as well.
=================
huh?
here are screen-shots of what it's doing on linux, which is what i expect
and desire:
http://smasher.org/tmp/snapshot149.png
http://smasher.org/tmp/snapshot150.png
here's a screen-shot of modifying IFS, which i almost thought would work,
but it makes sense that it doesn't work as above:
http://smasher.org/tmp/snapshot151.png
the 151 screen shot shows what doesn't work. if i try the commands used in
the 149 or 150 screen-shots on freebsd, i get the same non-results as
shown in 151; the name of the file i'm rsyncing, and then nothing.
> rsync --progress /from /to | sed -n l
>
> (or od -c)
>
> to see what rsync actually outputs.
================
or this:
<<<<<<<<
% rsync --progress -h --bwlimit=5 ./foo_in ./foo_out > test
% cat -vet test
foo_in$
32.77K 0% 0.00kB/s 0:00:00^M 65.54K 0% 5.00kB/s
0:49:49^M 98.30K 0% 5.00kB/s 0:49:42^M 131.07K 0%
5.00kB/s 0:49:36^M 163.84K 1% 5.00kB/s 0:49:29^M
196.61K 1% 5.00kB/s 0:49:22^M 229.38K 1% 5.00kB/s
0:49:16^M 262.14K 1% 5.00kB/s 0:49:09^M 294.91K 1%
5.00kB/s 0:49:03^M 327.68K 2% 5.00kB/s 0:48:56^M
360.45K 2% 5.00kB/s 0:48:50^M 393.22K 2% 5.00kB/s
0:48:43^M 425.98K 2% 5.00kB/s 0:48:36^M 458.75K 3%
5.00kB/s 0:48:30^M 491.52K 3% 5.00kB/s 0:48:24^M
<<snip>
<<<<<<<<
all of rsync's output of interest is stdout. it's mostly a bunch of
progress updates, separated by carriage returns.
--
...atom
________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------
"I am a great mayor;
I am an upstanding Christian man;
I am an intelligent man;
I am a deeply educated man;
I am a humble man."
-- Marion Barry
Mayor, Washington, D.C
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: freebsd problems with carriage return
2009-03-24 21:36 ` Atom Smasher
@ 2009-03-25 8:18 ` Benjamin R. Haskell
2009-03-25 12:23 ` Stephane Chazelas
2009-03-25 11:02 ` Stephane Chazelas
1 sibling, 1 reply; 8+ messages in thread
From: Benjamin R. Haskell @ 2009-03-25 8:18 UTC (permalink / raw)
To: Zsh Users
On Wed, 25 Mar 2009, Atom Smasher wrote:
> On Tue, 24 Mar 2009, Stephane Chazelas wrote:
>
> > [...]
> >
> > rsync --progress /from /to | sed -n l
> >
> > (or od -c)
> >
> > to see what rsync actually outputs.
> ================
>
> or this:
>
> <<<<<<<<
>
> % rsync --progress -h --bwlimit=5 ./foo_in ./foo_out > test
>
> % cat -vet test
> foo_in$
> 32.77K 0% 0.00kB/s 0:00:00^M 65.54K 0% 5.00kB/s
> 0:49:49^M 98.30K 0% 5.00kB/s 0:49:42^M 131.07K 0% 5.00kB/s
> 0:49:36^M 163.84K 1% 5.00kB/s 0:49:29^M 196.61K 1% 5.00kB/s
> 0:49:22^M 229.38K 1% 5.00kB/s 0:49:16^M 262.14K 1% 5.00kB/s
> 0:49:09^M 294.91K 1% 5.00kB/s 0:49:03^M 327.68K 2% 5.00kB/s
> 0:48:56^M 360.45K 2% 5.00kB/s 0:48:50^M 393.22K 2% 5.00kB/s
> 0:48:43^M 425.98K 2% 5.00kB/s 0:48:36^M 458.75K 3% 5.00kB/s
> 0:48:30^M 491.52K 3% 5.00kB/s 0:48:24^M
> <<snip>
>
> <<<<<<<<
>
> all of rsync's output of interest is stdout. it's mostly a bunch of progress
> updates, separated by carriage returns.
>
I think it's a difference in how FreeBSD and Linux handle some
tty/job-control settings. (Though it's not terribly surprising to me...
one of those "I'm going to behave differently when you pipe me" things.)
I assume your ultimate goal isn't to prepend 'foo' to each of those lines.
What are you trying to do?
On FreeBSD for me, the status updates are visible when not piped (either
displayed to terminal, or redirected to a file). But, rsync --progress
from to | cat -v only shows the final status.
I noticed in the rsync code (progress.c) that it declines to print the
partial status lines (the ones that end with '\r' so that they can be
overwritten) depending on the outcome of:
210 tc_pgrp = tcgetpgrp(STDOUT_FILENO);
211 if (tc_pgrp != pgrp && tc_pgrp != -1)
212 return;
I'm probably only noticing this because I just read:
"The TTY demystified"
http://www.linusakesson.net/programming/tty/index.php
Maybe something in 'stty' will affect the outcome of the above call (but
you'd have to ask someone who knows way more about it than I do). There
are at least a handful of differences in the various settings between
FreeBSD and Linux for the systems I'm using right now.
Best,
Ben
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: freebsd problems with carriage return
2009-03-24 21:36 ` Atom Smasher
2009-03-25 8:18 ` Benjamin R. Haskell
@ 2009-03-25 11:02 ` Stephane Chazelas
1 sibling, 0 replies; 8+ messages in thread
From: Stephane Chazelas @ 2009-03-25 11:02 UTC (permalink / raw)
To: Atom Smasher; +Cc: zsh-users
2009-03-25 10:36:21 +1300, Atom Smasher:
> On Tue, 24 Mar 2009, Stephane Chazelas wrote:
>
>> 2009-03-24 23:14:09 +1300, Atom Smasher:
>>> on linux this works as expected:
>>>
>>> rsync --progress /from /to | while read -d "`print '\r'`" n
>>
>> See $'\r' instead of "`print '\r'`"
>>
>>> do
>>> print " foo: $n"
>>
>> should be print -r, print without -r expands the \ sequences.
>>
>> You should probably use IFS= read -r instead of read as well.
> =================
>
> huh?
>
> here are screen-shots of what it's doing on linux, which is what i expect
> and desire:
> http://smasher.org/tmp/snapshot149.png
> http://smasher.org/tmp/snapshot150.png
>
> here's a screen-shot of modifying IFS, which i almost thought would work,
Sorry for the confusion, this was a remark on your code. I was
not suggesting that it was a fix for your problem.
read without -r processes the backslashes in its input
specially, so you generally need to pass it especially in
scripts.
Same, if you don't remove the white space characters from IFS,
read strips the leading and trailing ones.
$ print ' \\a \rb' | read -d $'\r' a; print -r "$a" | cat -vte
a$
$ print ' \\a \rb' | IFS= read -rd $'\r' a; print -r "$a" | cat -vte
\a $
[...]
>> rsync --progress /from /to | sed -n l
>>
>> (or od -c)
>>
>> to see what rsync actually outputs.
> ================
>
> or this:
>
> <<<<<<<<
>
> % rsync --progress -h --bwlimit=5 ./foo_in ./foo_out > test
>
> % cat -vet test
> foo_in$
> 32.77K 0% 0.00kB/s 0:00:00^M 65.54K 0% 5.00kB/s
> 0:49:49^M 98.30K 0% 5.00kB/s 0:49:42^M 131.07K 0%
> 5.00kB/s 0:49:36^M 163.84K 1% 5.00kB/s 0:49:29^M 196.61K
> 1% 5.00kB/s 0:49:22^M 229.38K 1% 5.00kB/s 0:49:16^M
> 262.14K 1% 5.00kB/s 0:49:09^M 294.91K 1% 5.00kB/s
> 0:49:03^M 327.68K 2% 5.00kB/s 0:48:56^M 360.45K 2%
> 5.00kB/s 0:48:50^M 393.22K 2% 5.00kB/s 0:48:43^M 425.98K
> 2% 5.00kB/s 0:48:36^M 458.75K 3% 5.00kB/s 0:48:30^M
> 491.52K 3% 5.00kB/s 0:48:24^M
> <<snip>
>
> <<<<<<<<
>
> all of rsync's output of interest is stdout. it's mostly a bunch of
> progress updates, separated by carriage returns.
>
[...]
But here, stdout is a regular file, not a pipe. Do you get the
same if you do rsync --progress -h --bwlimit=5 ./foo_in ./foo_out | cat -vet
Also, you could use ktrace/truss/strace or the equivalent on
your system to see if rsync is actually outputing those progress
lines.
--
Stephane
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: freebsd problems with carriage return
2009-03-25 8:18 ` Benjamin R. Haskell
@ 2009-03-25 12:23 ` Stephane Chazelas
2009-03-26 8:09 ` Atom Smasher
0 siblings, 1 reply; 8+ messages in thread
From: Stephane Chazelas @ 2009-03-25 12:23 UTC (permalink / raw)
To: Benjamin R. Haskell; +Cc: Zsh Users
2009-03-25 04:18:51 -0400, Benjamin R. Haskell:
[...]
> I think it's a difference in how FreeBSD and Linux handle some
> tty/job-control settings. (Though it's not terribly surprising to me...
> one of those "I'm going to behave differently when you pipe me" things.)
Except that as far as I can see from the source code, it does an
explicit fflush() after each progress line, so it shouldn't be a
stdio buffering issue. Unless the OP's using an old version of
rsync on FreeBSD?
[...]
> On FreeBSD for me, the status updates are visible when not piped (either
> displayed to terminal, or redirected to a file). But, rsync --progress
> from to | cat -v only shows the final status.
>
> I noticed in the rsync code (progress.c) that it declines to print the
> partial status lines (the ones that end with '\r' so that they can be
> overwritten) depending on the outcome of:
>
> 210 tc_pgrp = tcgetpgrp(STDOUT_FILENO);
> 211 if (tc_pgrp != pgrp && tc_pgrp != -1)
> 212 return;
The above is meant not to print the progress if the output is to
a terminal and rsync is not in the foreground process group of
the terminal.
For a pipe, tcgetpgrp should return -1 so that test should fail
(unless there's a bug in FreeBSD? ktrace would probably tell
us).
[...]
> Maybe something in 'stty' will affect the outcome of the above call (but
> you'd have to ask someone who knows way more about it than I do). There
> are at least a handful of differences in the various settings between
> FreeBSD and Linux for the systems I'm using right now.
[...]
It shouldn't have anything to do with stty as we're not
accessing the terminal here. And stty doesn't affect the
foreground process group of the terminal, it's the shell's job
control that does.
The fact that you don't get the output when stdout is a pipe is
probably the place to look at. But I don't know why rsync would
decide not to output the progress when stdout is pipe as opposed
to when it's a regular file.
Now, looking at:
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/sys_pipe.c?rev=1.201
pipe_ioctl(...
[...]
/* This is deprecated, FIOGETOWN should be used instead. */
case TIOCGPGRP:
*(int *)data = -fgetown(&mpipe->pipe_sigio);
break;
So tcgetpgrp() is valid on a pipe on FreeBSD, so it contradicts
its man page.
That would be the reason. To fix that, you'd have to patch rsync
to add a isatty() check, or better, remove that check
altogether, as if the user reguested progress with --progress, I
don't agree that rsync should disable it in any circumstance.
--
Stephane
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: freebsd problems with carriage return
2009-03-25 12:23 ` Stephane Chazelas
@ 2009-03-26 8:09 ` Atom Smasher
2009-03-26 9:45 ` Stephane Chazelas
0 siblings, 1 reply; 8+ messages in thread
From: Atom Smasher @ 2009-03-26 8:09 UTC (permalink / raw)
To: Stephane Chazelas; +Cc: Zsh Users
On Wed, 25 Mar 2009, Stephane Chazelas wrote:
> Except that as far as I can see from the source code, it does an
> explicit fflush() after each progress line, so it shouldn't be a stdio
> buffering issue. Unless the OP's using an old version of rsync on
> FreeBSD?
==================
freebsd:
rsync version 3.0.2
rsync version 3.0.5
linux:
rsync version 2.6.9
> The fact that you don't get the output when stdout is a pipe is probably
> the place to look at. But I don't know why rsync would decide not to
> output the progress when stdout is pipe as opposed to when it's a
> regular file.
====================
hhmmm... it may be related to a pipe (but it works as expected on linux,
with a pipe or redirect)... so maybe i can use a redirect as if it's a
pipe...
http://smasher.org/tmp/snapshot154.png
the top half of that split screen is `tr` converting carriage returns into
newlines, and reading stdin from a fifo. the bottom half is rsync, with
stdout redirected into the fifo. it's working as desired and expected,
even on freebsd.
what seems to me like just a slight variation of that, once again works on
linux and fails on freebsd:
tr '\r' '\n' < <( rsync --progress --bwlimit=5 foo_in foo_out )
or:
rsync --progress --bwlimit=5 foo_in foo_out > >( tr '\r' '\n' )
my understanding of zsh; process substitution should operate as a fifo. so
i really don't get why it works as desired in the screen-shot, but not
with process substitution.
--
...atom
________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------
"Only when the last tree has died,
And the last river poisoned,
And the last fish been caught,
Will the white man realize that he cannot eat money"
--19th Century Cree Indian Proverb
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: freebsd problems with carriage return
2009-03-26 8:09 ` Atom Smasher
@ 2009-03-26 9:45 ` Stephane Chazelas
0 siblings, 0 replies; 8+ messages in thread
From: Stephane Chazelas @ 2009-03-26 9:45 UTC (permalink / raw)
To: Atom Smasher; +Cc: Zsh Users
2009-03-26 21:09:58 +1300, Atom Smasher:
[...]
> hhmmm... it may be related to a pipe (but it works as expected on linux,
> with a pipe or redirect)... so maybe i can use a redirect as if it's a
> pipe...
> http://smasher.org/tmp/snapshot154.png
[...]
Sorry, my previous email was a bit convoluted as I was writing
it while investigating. The last part of the email explained the
reason for the problem. On FreeBSD, tcgetpgrp() on a pipe
returns the id of the process group that is meant to receive a
SIGIO on that pipe (or 0 if nobody registered for SIGIO) while
rsync() expects it to return -1 (as it should according to POSIX
and its man page).
The effect of that is that as tcgetpgrp() return 0 instead of
-1, rsync things stdout is a terminal and as 0 is not the
progress group id of the process, it thinks it is running in
background so does *not* output its progress.
>
> the top half of that split screen is `tr` converting carriage returns into
> newlines, and reading stdin from a fifo. the bottom half is rsync, with
> stdout redirected into the fifo. it's working as desired and expected, even
> on freebsd.
>
> what seems to me like just a slight variation of that, once again works on
> linux and fails on freebsd:
> tr '\r' '\n' < <( rsync --progress --bwlimit=5 foo_in foo_out )
> or:
> rsync --progress --bwlimit=5 foo_in foo_out > >( tr '\r' '\n' )
>
> my understanding of zsh; process substitution should operate as a fifo. so
> i really don't get why it works as desired in the screen-shot, but not with
> process substitution.
[...]
Same thing, it's all pipes, so falls into that case.
Cheers,
Stephane
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2009-03-26 9:46 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-03-24 10:14 freebsd problems with carriage return Atom Smasher
2009-03-24 11:47 ` Stephane Chazelas
2009-03-24 21:36 ` Atom Smasher
2009-03-25 8:18 ` Benjamin R. Haskell
2009-03-25 12:23 ` Stephane Chazelas
2009-03-26 8:09 ` Atom Smasher
2009-03-26 9:45 ` Stephane Chazelas
2009-03-25 11:02 ` Stephane Chazelas
Code repositories for project(s) associated with this public inbox
https://git.vuxu.org/mirror/zsh/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).