freebsd problems with carriage return

freebsd problems with carriage return

[Atom Smasher]

on linux this works as expected:

     rsync --progress /from /to | while read -d "`print '\r'`" n
     do
         print "  foo: $n"
     done

as rsync uses a carriage return to update the progress of the file 
transfer, the "read" builtin uses '\r' as a delimiter, and prints the 
output as expected, one line at a time.

on freebsd it doesn't work at all. i suspect the bug may be in bsd's 
libraries, because a similar thing happens on linux and fails to happen on 
freebsd with:
     rsync --progress /from /to | tr '\r' '\n'

of course linux is using gnu-tr and freebsd is using bsd-tr.

can anyone help identify the source of the problem, and/or a workaround?

thanks...!


-- 
         ...atom

  ________________________
  http://atom.smasher.org/
  762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
  -------------------------------------------------

 	"War will cease when men refuse to fight."
 		-- F. Hansen

Re: freebsd problems with carriage return

[Stephane Chazelas]

2009-03-24 23:14:09 +1300, Atom Smasher:
> on linux this works as expected:
> 
>      rsync --progress /from /to | while read -d "`print '\r'`" n

See $'\r' instead of "`print '\r'`"

>      do
>          print "  foo: $n"

should be print -r, print without -r expands the \ sequences.

You should probably use IFS= read -r instead of read as well.

>      done
> 
> as rsync uses a carriage return to update the progress of the file 
> transfer, the "read" builtin uses '\r' as a delimiter, and prints the 
> output as expected, one line at a time.
> 
> on freebsd it doesn't work at all.

In which way does it not work?

> i suspect the bug may be in bsd's 
> libraries, because a similar thing happens on linux and fails to happen on 
> freebsd with:
>      rsync --progress /from /to | tr '\r' '\n'
> 
> of course linux is using gnu-tr and freebsd is using bsd-tr.
> 
> can anyone help identify the source of the problem, and/or a workaround?
[...]

run

rsync --progress /from /to | sed -n l

(or od -c)

to see what rsync actually outputs.

Also, in case rsync outputs it on stderr, try:

rsync --progress /from /to 2>&1 | sed -n l

-- 
Stephane

Re: freebsd problems with carriage return

[Atom Smasher]

On Tue, 24 Mar 2009, Stephane Chazelas wrote:

> 2009-03-24 23:14:09 +1300, Atom Smasher:
>> on linux this works as expected:
>>
>>      rsync --progress /from /to | while read -d "`print '\r'`" n
>
> See $'\r' instead of "`print '\r'`"
>
>>      do
>>          print "  foo: $n"
>
> should be print -r, print without -r expands the \ sequences.
>
> You should probably use IFS= read -r instead of read as well.
=================

huh?

here are screen-shots of what it's doing on linux, which is what i expect 
and desire:
 	http://smasher.org/tmp/snapshot149.png
 	http://smasher.org/tmp/snapshot150.png

here's a screen-shot of modifying IFS, which i almost thought would work, 
but it makes sense that it doesn't work as above:
 	http://smasher.org/tmp/snapshot151.png

the 151 screen shot shows what doesn't work. if i try the commands used in 
the 149 or 150 screen-shots on freebsd, i get the same non-results as 
shown in 151; the name of the file i'm rsyncing, and then nothing.


> rsync --progress /from /to | sed -n l
>
> (or od -c)
>
> to see what rsync actually outputs.
================

or this:

<<<<<<<<

% rsync --progress -h --bwlimit=5 ./foo_in ./foo_out > test

% cat -vet test
foo_in$
       32.77K   0%    0.00kB/s    0:00:00^M      65.54K   0%    5.00kB/s 
0:49:49^M      98.30K   0%    5.00kB/s    0:49:42^M     131.07K   0% 
5.00kB/s    0:49:36^M     163.84K   1%    5.00kB/s    0:49:29^M 
196.61K   1%    5.00kB/s    0:49:22^M     229.38K   1%    5.00kB/s 
0:49:16^M     262.14K   1%    5.00kB/s    0:49:09^M     294.91K   1% 
5.00kB/s    0:49:03^M     327.68K   2%    5.00kB/s    0:48:56^M 
360.45K   2%    5.00kB/s    0:48:50^M     393.22K   2%    5.00kB/s 
0:48:43^M     425.98K   2%    5.00kB/s    0:48:36^M     458.75K   3% 
5.00kB/s    0:48:30^M     491.52K   3%    5.00kB/s    0:48:24^M
 	<<snip>

<<<<<<<<

all of rsync's output of interest is stdout. it's mostly a bunch of 
progress updates, separated by carriage returns.


-- 
         ...atom

  ________________________
  http://atom.smasher.org/
  762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
  -------------------------------------------------

 	"I am a great mayor;
 	 I am an upstanding Christian man;
 	 I am an intelligent man;
 	 I am a deeply educated man;
 	 I am a humble man."
 		-- Marion Barry
 		Mayor, Washington, D.C

Re: freebsd problems with carriage return

[Benjamin R. Haskell]

On Wed, 25 Mar 2009, Atom Smasher wrote:

> On Tue, 24 Mar 2009, Stephane Chazelas wrote:
> 
> > [...]
> >
> > rsync --progress /from /to | sed -n l
> >
> > (or od -c)
> >
> > to see what rsync actually outputs.
> ================
> 
> or this:
> 
> <<<<<<<<
> 
> % rsync --progress -h --bwlimit=5 ./foo_in ./foo_out > test
> 
> % cat -vet test
> foo_in$
>       32.77K   0%    0.00kB/s    0:00:00^M      65.54K   0%    5.00kB/s
> 0:49:49^M      98.30K   0%    5.00kB/s    0:49:42^M     131.07K   0% 5.00kB/s
> 0:49:36^M     163.84K   1%    5.00kB/s    0:49:29^M 196.61K   1%    5.00kB/s
> 0:49:22^M     229.38K   1%    5.00kB/s 0:49:16^M     262.14K   1%    5.00kB/s
> 0:49:09^M     294.91K   1% 5.00kB/s    0:49:03^M     327.68K   2%    5.00kB/s
> 0:48:56^M 360.45K   2%    5.00kB/s    0:48:50^M     393.22K   2%    5.00kB/s
> 0:48:43^M     425.98K   2%    5.00kB/s    0:48:36^M     458.75K   3% 5.00kB/s
> 0:48:30^M     491.52K   3%    5.00kB/s    0:48:24^M
> 	<<snip>
> 
> <<<<<<<<
> 
> all of rsync's output of interest is stdout. it's mostly a bunch of progress
> updates, separated by carriage returns.
> 

I think it's a difference in how FreeBSD and Linux handle some 
tty/job-control settings.  (Though it's not terribly surprising to me... 
one of those "I'm going to behave differently when you pipe me" things.)  
I assume your ultimate goal isn't to prepend 'foo' to each of those lines.  
What are you trying to do?

On FreeBSD for me, the status updates are visible when not piped (either 
displayed to terminal, or redirected to a file).  But, rsync --progress 
from to | cat -v only shows the final status.

I noticed in the rsync code (progress.c) that it declines to print the 
partial status lines (the ones that end with '\r' so that they can be 
overwritten) depending on the outcome of:

 210         tc_pgrp = tcgetpgrp(STDOUT_FILENO);
 211         if (tc_pgrp != pgrp && tc_pgrp != -1)
 212                 return;


I'm probably only noticing this because I just read:
"The TTY demystified"
http://www.linusakesson.net/programming/tty/index.php

Maybe something in 'stty' will affect the outcome of the above call (but 
you'd have to ask someone who knows way more about it than I do).  There 
are at least a handful of differences in the various settings between 
FreeBSD and Linux for the systems I'm using right now.

Best,
Ben

Re: freebsd problems with carriage return

[Stephane Chazelas]

2009-03-25 10:36:21 +1300, Atom Smasher:
> On Tue, 24 Mar 2009, Stephane Chazelas wrote:
>
>> 2009-03-24 23:14:09 +1300, Atom Smasher:
>>> on linux this works as expected:
>>>
>>>      rsync --progress /from /to | while read -d "`print '\r'`" n
>>
>> See $'\r' instead of "`print '\r'`"
>>
>>>      do
>>>          print "  foo: $n"
>>
>> should be print -r, print without -r expands the \ sequences.
>>
>> You should probably use IFS= read -r instead of read as well.
> =================
>
> huh?
>
> here are screen-shots of what it's doing on linux, which is what i expect 
> and desire:
> 	http://smasher.org/tmp/snapshot149.png
> 	http://smasher.org/tmp/snapshot150.png
>
> here's a screen-shot of modifying IFS, which i almost thought would work, 

Sorry for the confusion, this was a remark on your code. I was
not suggesting that it was a fix for your problem.

read without -r processes the backslashes in its input
specially, so you generally need to pass it especially in
scripts.

Same, if you don't remove the white space characters from IFS,
read strips the leading and trailing ones.

$ print '  \\a \rb' | read -d $'\r' a; print -r "$a" | cat -vte
a$
$ print '  \\a \rb' | IFS= read -rd $'\r' a; print -r "$a" | cat -vte
  \a $

[...]
>> rsync --progress /from /to | sed -n l
>>
>> (or od -c)
>>
>> to see what rsync actually outputs.
> ================
>
> or this:
>
> <<<<<<<<
>
> % rsync --progress -h --bwlimit=5 ./foo_in ./foo_out > test
>
> % cat -vet test
> foo_in$
>       32.77K   0%    0.00kB/s    0:00:00^M      65.54K   0%    5.00kB/s 
> 0:49:49^M      98.30K   0%    5.00kB/s    0:49:42^M     131.07K   0% 
> 5.00kB/s    0:49:36^M     163.84K   1%    5.00kB/s    0:49:29^M 196.61K   
> 1%    5.00kB/s    0:49:22^M     229.38K   1%    5.00kB/s 0:49:16^M     
> 262.14K   1%    5.00kB/s    0:49:09^M     294.91K   1% 5.00kB/s    
> 0:49:03^M     327.68K   2%    5.00kB/s    0:48:56^M 360.45K   2%    
> 5.00kB/s    0:48:50^M     393.22K   2%    5.00kB/s 0:48:43^M     425.98K   
> 2%    5.00kB/s    0:48:36^M     458.75K   3% 5.00kB/s    0:48:30^M     
> 491.52K   3%    5.00kB/s    0:48:24^M
> 	<<snip>
>
> <<<<<<<<
>
> all of rsync's output of interest is stdout. it's mostly a bunch of 
> progress updates, separated by carriage returns.
>
[...]

But here, stdout is a regular file, not a pipe. Do you get the
same if you do rsync --progress -h --bwlimit=5 ./foo_in ./foo_out | cat -vet

Also, you could use ktrace/truss/strace or the equivalent on
your system to see if rsync is actually outputing those progress
lines.

-- 
Stephane

Re: freebsd problems with carriage return

[Stephane Chazelas]

2009-03-25 04:18:51 -0400, Benjamin R. Haskell:
[...]
> I think it's a difference in how FreeBSD and Linux handle some 
> tty/job-control settings.  (Though it's not terribly surprising to me... 
> one of those "I'm going to behave differently when you pipe me" things.)  

Except that as far as I can see from the source code, it does an
explicit fflush() after each progress line, so it shouldn't be a
stdio buffering issue. Unless the OP's using an old version of
rsync on FreeBSD?

[...]
> On FreeBSD for me, the status updates are visible when not piped (either 
> displayed to terminal, or redirected to a file).  But, rsync --progress 
> from to | cat -v only shows the final status.
> 
> I noticed in the rsync code (progress.c) that it declines to print the 
> partial status lines (the ones that end with '\r' so that they can be 
> overwritten) depending on the outcome of:
> 
>  210         tc_pgrp = tcgetpgrp(STDOUT_FILENO);
>  211         if (tc_pgrp != pgrp && tc_pgrp != -1)
>  212                 return;

The above is meant not to print the progress if the output is to
a terminal and rsync is not in the foreground process group of
the terminal.

For a pipe, tcgetpgrp should return -1 so that test should fail
(unless there's a bug in FreeBSD? ktrace would probably tell
us).

[...]
> Maybe something in 'stty' will affect the outcome of the above call (but 
> you'd have to ask someone who knows way more about it than I do).  There 
> are at least a handful of differences in the various settings between 
> FreeBSD and Linux for the systems I'm using right now.
[...]

It shouldn't have anything to do with stty as we're not
accessing the terminal here. And stty doesn't affect the
foreground process group of the terminal, it's the shell's job
control that does.

The fact that you don't get the output when stdout is a pipe is
probably the place to look at. But I don't know why rsync would
decide not to output the progress when stdout is pipe as opposed
to when it's a regular file.

Now, looking at:
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/sys_pipe.c?rev=1.201

pipe_ioctl(...
[...]
         /* This is deprecated, FIOGETOWN should be used instead. */
         case TIOCGPGRP:
                 *(int *)data = -fgetown(&mpipe->pipe_sigio);
                 break;

So tcgetpgrp() is valid on a pipe on FreeBSD, so it contradicts
its man page.

That would be the reason. To fix that, you'd have to patch rsync
to add a isatty() check, or better, remove that check
altogether, as if the user reguested progress with --progress, I
don't agree that rsync should disable it in any circumstance.

-- 
Stephane

Re: freebsd problems with carriage return

[Atom Smasher]

On Wed, 25 Mar 2009, Stephane Chazelas wrote:

> Except that as far as I can see from the source code, it does an 
> explicit fflush() after each progress line, so it shouldn't be a stdio 
> buffering issue. Unless the OP's using an old version of rsync on 
> FreeBSD?
==================

freebsd:
 	rsync  version 3.0.2
 	rsync  version 3.0.5
linux:
 	rsync  version 2.6.9


> The fact that you don't get the output when stdout is a pipe is probably 
> the place to look at. But I don't know why rsync would decide not to 
> output the progress when stdout is pipe as opposed to when it's a 
> regular file.
====================

hhmmm... it may be related to a pipe (but it works as expected on linux, 
with a pipe or redirect)... so maybe i can use a redirect as if it's a 
pipe...
 	http://smasher.org/tmp/snapshot154.png

the top half of that split screen is `tr` converting carriage returns into 
newlines, and reading stdin from a fifo. the bottom half is rsync, with 
stdout redirected into the fifo. it's working as desired and expected, 
even on freebsd.

what seems to me like just a slight variation of that, once again works on 
linux and fails on freebsd:
 	tr '\r' '\n' < <( rsync --progress --bwlimit=5 foo_in foo_out )
  or:
 	rsync --progress --bwlimit=5 foo_in foo_out > >( tr '\r' '\n' )

my understanding of zsh; process substitution should operate as a fifo. so 
i really don't get why it works as desired in the screen-shot, but not 
with process substitution.


-- 
         ...atom

  ________________________
  http://atom.smasher.org/
  762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
  -------------------------------------------------

 	"Only when the last tree has died,
 	 And the last river poisoned,
 	 And the last fish been caught,
 	 Will the white man realize that he cannot eat money"
 		--19th Century Cree Indian Proverb

Re: freebsd problems with carriage return

[Stephane Chazelas]

2009-03-26 21:09:58 +1300, Atom Smasher:
[...]
> hhmmm... it may be related to a pipe (but it works as expected on linux, 
> with a pipe or redirect)... so maybe i can use a redirect as if it's a 
> pipe...
> 	http://smasher.org/tmp/snapshot154.png
[...]

Sorry, my previous email was a bit convoluted as I was writing
it while investigating. The last part of the email explained the
reason for the problem. On FreeBSD, tcgetpgrp() on a pipe
returns the id of the process group that is meant to receive a
SIGIO on that pipe (or 0 if nobody registered for SIGIO) while
rsync() expects it to return -1 (as it should according to POSIX
and its man page).

The effect of that is that as tcgetpgrp() return 0 instead of
-1, rsync things stdout is a terminal and as 0 is not the
progress group id of the process, it thinks it is running in
background so does *not* output its progress.

>
> the top half of that split screen is `tr` converting carriage returns into 
> newlines, and reading stdin from a fifo. the bottom half is rsync, with 
> stdout redirected into the fifo. it's working as desired and expected, even 
> on freebsd.
>
> what seems to me like just a slight variation of that, once again works on 
> linux and fails on freebsd:
> 	tr '\r' '\n' < <( rsync --progress --bwlimit=5 foo_in foo_out )
>  or:
> 	rsync --progress --bwlimit=5 foo_in foo_out > >( tr '\r' '\n' )
>
> my understanding of zsh; process substitution should operate as a fifo. so 
> i really don't get why it works as desired in the screen-shot, but not with 
> process substitution.
[...]

Same thing, it's all pipes, so falls into that case.

Cheers,
Stephane