pure zsh implementation of wget

pure zsh implementation of wget

[Guido van Steen]

Hi! 

I was wondering if someone could point me to a pure zsh implementation of a function/script that downloads files like wget. 

I have been look at this (slightly modified from http://rosettacode.org/wiki/HTTP#Zsh): 

$cat wget.zsh
#!/usr/bin/env zsh
zmodload zsh/net/tcp
ztcp google.com 80
fd=$REPLY
print -l -u $fd -- 'GET / HTTP/1.1' 'Host: google.com' ''
while read -u $fd -r -e -t 1; do; :; done
ztcp -c $fd

However, when I run this script, I do not receive any files. Neither do I observe any other results. I tried it with other sites, e.g. replacing google.com by localhost, and so on. 

Does someone know what am I missing? 

BTW, would this work with https as well? 

Thanks in advance! 

Guido 

Re: pure zsh implementation of wget

[Mikael Magnusson]

On 22 September 2011 23:06, Guido van Steen <gvsteen@yahoo.com> wrote:
> Hi!
>
> I was wondering if someone could point me to a pure zsh implementation of a function/script that downloads files like wget.
>
> I have been look at this (slightly modified from http://rosettacode.org/wiki/HTTP#Zsh):
>
> $cat wget.zsh
> #!/usr/bin/env zsh
> zmodload zsh/net/tcp
> ztcp google.com 80
> fd=$REPLY
> print -l -u $fd -- 'GET / HTTP/1.1' 'Host: google.com' ''
> while read -u $fd -r -e -t 1; do; :; done
> ztcp -c $fd
>
> However, when I run this script, I do not receive any files. Neither do I observe any other results. I tried it with other sites, e.g. replacing google.com by localhost, and so on.
> Does someone know what am I missing?

It works for me, but there are a couple of problems. It uses a timeout
of one second, so if your connection is slow it won't do anything, and
it doesn't include the dos newlines i believe you should have in http.
A more severe problem is that it uses read without setting IFS empty
which means it will mangle initial whitespace input. All these issues
(but possibly not others) are solved here (possibly):

#!/bin/zsh
zmodload zsh/net/tcp
ztcp google.com 80
fd=$REPLY
print -l -u $fd -- 'GET / HTTP/1.1'$'\015' 'Host:
www.google.com'$'\015' 'Connection: close'$'\015' $'\015'
while IFS= read -u $fd -r -e; do; :; done
ztcp -c $fd

note that gmail will insert a newline in the print line, so join that
back yourself :) Instead of the while loop you can also just write cat
<&$fd

> BTW, would this work with https as well?

Definitely not :).

-- 
Mikael Magnusson

Re: pure zsh implementation of wget

[Guido van Steen]

Thanks a lot! With these additions it also works for me. 

How difficult would it be to make this handle https as well? 

Is there any existing implementation that handles https? 

Best wishes, 

Guido 

--- On Fri, 23/9/11, Mikael Magnusson <mikachu@gmail.com> wrote:

> From: Mikael Magnusson <mikachu@gmail.com>
> Subject: Re: pure zsh implementation of wget
> To: "Guido van Steen" <gvsteen@yahoo.com>
> Cc: zsh-users@zsh.org
> Date: Friday, 23 September, 2011, 4:50 AM
> On 22 September 2011 23:06, Guido van
> Steen <gvsteen@yahoo.com>
> wrote:
> > Hi!
> >
> > I was wondering if someone could point me to a pure
> zsh implementation of a function/script that downloads files
> like wget.
> >
> > I have been look at this (slightly modified from http://rosettacode.org/wiki/HTTP#Zsh):
> >
> > $cat wget.zsh
> > #!/usr/bin/env zsh
> > zmodload zsh/net/tcp
> > ztcp google.com 80
> > fd=$REPLY
> > print -l -u $fd -- 'GET / HTTP/1.1' 'Host: google.com'
> ''
> > while read -u $fd -r -e -t 1; do; :; done
> > ztcp -c $fd
> >
> > However, when I run this script, I do not receive any
> files. Neither do I observe any other results. I tried it
> with other sites, e.g. replacing google.com by localhost,
> and so on.
> > Does someone know what am I missing?
> 
> It works for me, but there are a couple of problems. It
> uses a timeout
> of one second, so if your connection is slow it won't do
> anything, and
> it doesn't include the dos newlines i believe you should
> have in http.
> A more severe problem is that it uses read without setting
> IFS empty
> which means it will mangle initial whitespace input. All
> these issues
> (but possibly not others) are solved here (possibly):
> 
> #!/bin/zsh
> zmodload zsh/net/tcp
> ztcp google.com 80
> fd=$REPLY
> print -l -u $fd -- 'GET / HTTP/1.1'$'\015' 'Host:
> www.google.com'$'\015' 'Connection: close'$'\015' $'\015'
> while IFS= read -u $fd -r -e; do; :; done
> ztcp -c $fd
> 
> note that gmail will insert a newline in the print line, so
> join that
> back yourself :) Instead of the while loop you can also
> just write cat
> <&$fd
> 
> > BTW, would this work with https as well?
> 
> Definitely not :).
> 
> -- 
> Mikael Magnusson
>

Re: pure zsh implementation of wget

[Mikael Magnusson]

On 23 September 2011 07:23, Guido van Steen <gvsteen@yahoo.com> wrote:
> Thanks a lot! With these additions it also works for me.
>
> How difficult would it be to make this handle https as well?
>
> Is there any existing implementation that handles https?

You would probably need to make a zsh module for it using some library
like openssl or gnutls. I suppose you could implement ssl in pure
shell, but... why?

-- 
Mikael Magnusson

Re: pure zsh implementation of wget

[Guido van Steen]

I would like to avoid depending on wget (for a package I maintain). 

Never mind, I was only wondering if it could easily be done in pure zsh. So, the answer is: "it won't be that easy"... 

--- On Fri, 23/9/11, Mikael Magnusson <mikachu@gmail.com> wrote:

> From: Mikael Magnusson <mikachu@gmail.com>
> Subject: Re: pure zsh implementation of wget
> To: "Guido van Steen" <gvsteen@yahoo.com>
> Cc: zsh-users@zsh.org
> Date: Friday, 23 September, 2011, 1:57 PM
> On 23 September 2011 07:23, Guido van
> Steen <gvsteen@yahoo.com>
> wrote:
> > Thanks a lot! With these additions it also works for
> me.
> >
> > How difficult would it be to make this handle https as
> well?
> >
> > Is there any existing implementation that handles
> https?
> 
> You would probably need to make a zsh module for it using
> some library
> like openssl or gnutls. I suppose you could implement ssl
> in pure
> shell, but... why?
> 
> -- 
> Mikael Magnusson
>

Re: pure zsh implementation of wget

[Guillaume Brunerie]

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]

2011/9/23 Guido van Steen <gvsteen@yahoo.com>

> I would like to avoid depending on wget (for a package I maintain).
>

You do not want to depend on wget but don’t mind depending on zsh instead?
Are there any Linux distribution / UNIX-like operating system without wget
installed by default? (and with zsh by default)

Never mind, I was only wondering if it could easily be done in pure zsh. So,
> the answer is: "it won't be that easy"...
>
> --- On Fri, 23/9/11, Mikael Magnusson <mikachu@gmail.com> wrote:
>
> > From: Mikael Magnusson <mikachu@gmail.com>
> > Subject: Re: pure zsh implementation of wget
> > To: "Guido van Steen" <gvsteen@yahoo.com>
> > Cc: zsh-users@zsh.org
> > Date: Friday, 23 September, 2011, 1:57 PM
> > On 23 September 2011 07:23, Guido van
> > Steen <gvsteen@yahoo.com>
> > wrote:
> > > Thanks a lot! With these additions it also works for
> > me.
> > >
> > > How difficult would it be to make this handle https as
> > well?
> > >
> > > Is there any existing implementation that handles
> > https?
> >
> > You would probably need to make a zsh module for it using
> > some library
> > like openssl or gnutls. I suppose you could implement ssl
> > in pure
> > shell, but... why?
> >
> > --
> > Mikael Magnusson
> >
>

Re: pure zsh implementation of wget

[İsmail Dönmez]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

Hi;

On Fri, Sep 23, 2011 at 2:06 PM, Guillaume Brunerie <
guillaume.brunerie@gmail.com> wrote:

> 2011/9/23 Guido van Steen <gvsteen@yahoo.com>
>
> > I would like to avoid depending on wget (for a package I maintain).
> >
>
> You do not want to depend on wget but don’t mind depending on zsh instead?
> Are there any Linux distribution / UNIX-like operating system without wget
> installed by default? (and with zsh by default)
>

OSX doesn't ship any wget but zsh by default. But there is curl available,
so...

Regards,
ismail

Re: pure zsh implementation of wget

[Bart Schaefer]

On Sep 22, 10:23pm, Guido van Steen wrote:
}
} How difficult would it be to make this handle https as well? 

It'd be fairly difficult.  You'd need a new zsh module that links
libopenssl into the shell, to start with.

} Is there any existing implementation that handles https? 

Existing implementation of what?

Re: pure zsh implementation of wget

[Guido van Steen]

Thanks Bart, 

I meant a zsh web client capable of handling https. It seems like no one has ever tried it. So, I won't try either :)

Best wishes, 

Guido 


--- On Fri, 23/9/11, Bart Schaefer <schaefer@brasslantern.com> wrote:

> From: Bart Schaefer <schaefer@brasslantern.com>
> Subject: Re: pure zsh implementation of wget
> To: zsh-users@zsh.org
> Date: Friday, 23 September, 2011, 2:01 PM
> On Sep 22, 10:23pm, Guido van Steen
> wrote:
> }
> } How difficult would it be to make this handle https as
> well? 
> 
> It'd be fairly difficult.  You'd need a new zsh module
> that links
> libopenssl into the shell, to start with.
> 
> } Is there any existing implementation that handles https?
> 
> 
> Existing implementation of what?
>

Re: pure zsh implementation of wget

[Oliver Kiddle]

On 22 Sep, Guido van Steen wrote:
> Thanks a lot! With these additions it also works for me. 
> 
> How difficult would it be to make this handle https as well? 

You can easily run openssl or stunnel in a coprocess to handle the SSL
for you: openssl s_client -quiet -connect host:443

But this doesn't really meet your requirement of pure zsh: you might
aswell be running wget.

Oliver

Re: pure zsh implementation of wget

[Guido van Steen]

Hi, 

One more question: How should I modify that snippet to download a random  url, such as like http://security.debian.org/pool/updates/main/i/isc-dhcp/dhcp3-client_4.1.1-P1-15+squeeze3_all.deb

Thanks in advance! 

Guido 

> #!/bin/zsh
> zmodload zsh/net/tcp
> ztcp google.com 80
> fd=$REPLY
> print -l -u $fd -- 'GET / HTTP/1.1'$'\015' 'Host:
> www.google.com'$'\015' 'Connection: close'$'\015' $'\015'
> while IFS= read -u $fd -r -e; do; :; done
> ztcp -c $fd

Re: pure zsh implementation of wget

[Bart Schaefer]

On Sep 24,  9:49am, Guido van Steen wrote:
}
} One more question: How should I modify that snippet to download a
} random url, such as like
} http://security.debian.org/pool/updates/main/i/isc-dhcp/dhcp3-client_4.1.1-P1-15+squeeze3_all.deb

I think you mean "arbitrary" rather than "random"?

Which in turn means you want to parse the URL.  A quick way is probably

  IFS=/ read scheme empty server resource <<<$theurl

which leads to

zwget() {
    emulate -LR zsh
    local scheme empty server resource fd
    IFS=/ read scheme empty server resource <<<$1
    case $scheme in
    (https:) print -u2 SSL unsupported, falling back on HTTP ;&
    (http:)
	zmodload zsh/net/tcp
	ztcp $server 80 && fd=$REPLY || exit 1;;
    (*) print -u2 $scheme unsupported; exit 1;;
    esac
    print -l -u$fd -- \
	"GET /$resource HTTP/1.1"$'\015' \
	"Host: $server"$'\015' \
	'Connection: close'$'\015' $'\015'
    while IFS= read -u $fd -r -e; do; :; done
    ztcp -c $fd
}

You can probably work out how to support ftp: using the zsh/zftp module.

Re: pure zsh implementation of wget

[Guido van Steen]

Thank you, Bart! 

Now hopefully my last question: is there an easy way to get rid of the first part of the file, which starts with: 

HTTP/1.1 200 OK

Date: Sat, 24 Sep 2011 20:06:24 GMT

Server: Apache

Last-Modified: Thu, 11 Aug 2011 05:26:04 GMT

ETag: "6378-4aa340878d300"

Accept-Ranges: bytes

Content-Length: 25464

Connection: close

Content-Type: application/x-debian-package



The downloaded file should start with: 

!<arch>
debian-binary   1312914433  0     0     100644  4         `
2.0
control.tar.gz  1312914433  0     0     100644  911       `
...
...  

Best wishes, 

Guido

Re: pure zsh implementation of wget

[Guido van Steen]

Hi, 

Well, sorry for answering my own question: I just thought of using tail, and then getting rid of the superfluous <eof> through head. 

% tail dhcp3-client_4.1.1-P1-15+squeeze3_all.deb -c 25465 > dhcp3-client_4.1.1-P1-15+squeeze3_all.deb 

% head dhcp3-client_4.1.1-P1-15+squeeze3_all.deb -c -1 > dhcp3-client_4.1.1-P1-15+squeeze3_all.deb

It is a bit ugly but it seems to work. 

Best wishes, 

Guido 

--- On Sun, 25/9/11, Guido van Steen <gvsteen@yahoo.com> wrote:

> From: Guido van Steen <gvsteen@yahoo.com>
> Subject: Re: pure zsh implementation of wget
> To: "Bart Schaefer" <schaefer@brasslantern.com>
> Cc: zsh-users@zsh.org
> Date: Sunday, 25 September, 2011, 3:23 AM
> Thank you, Bart! 
> 
> Now hopefully my last question: is there an easy way to get
> rid of the first part of the file, which starts with: 
> 
> HTTP/1.1 200 OK
> 
> Date: Sat, 24 Sep 2011 20:06:24 GMT
> 
> Server: Apache
> 
> Last-Modified: Thu, 11 Aug 2011 05:26:04 GMT
> 
> ETag: "6378-4aa340878d300"
> 
> Accept-Ranges: bytes
> 
> Content-Length: 25464
> 
> Connection: close
> 
> Content-Type: application/x-debian-package
> 
> 
> 
> The downloaded file should start with: 
> 
> !<arch>
> debian-binary   1312914433  0 
>    0     100644 
> 4         `
> 2.0
> control.tar.gz  1312914433  0 
>    0     100644 
> 911       `
> ...
> ...  
> 
> Best wishes, 
> 
> Guido
>

Re: pure zsh implementation of wget

[Bart Schaefer]

On Sep 24,  1:23pm, Guido van Steen wrote:
}
} Now hopefully my last question: is there an easy way to get rid of the
} first part of the file

That's just the standard header in the HTTP format, like the header on
an email message.  You just have to discard everything up to the first
blank line.

One other thing you may want to do is switch from 'HTTP/1.1' to 1.0 to
prevent the server from breaking the content into chunks.  You'll be
limited to 10MB downloads but won't have to interpret any content after
the first blank line.

A blank line here means one matching $'\015\012', but "read" will strip
the $'\012' so:

zwget() {
    emulate -LR zsh
    local scheme empty server resource fd headerline
    IFS=/ read scheme empty server resource <<<$1
    case $scheme in
    (https:) print -u2 SSL unsupported, falling back on HTTP ;&
    (http:)
        zmodload zsh/net/tcp
        ztcp $server 80 && fd=$REPLY || return 1;;
    (*) print -u2 $scheme unsupported; return 1;;
    esac
    print -l -u$fd -- \
        "GET /$resource HTTP/1.0"$'\015' \
        "Host: $server"$'\015' \
        'Connection: close'$'\015' $'\015'
    while IFS= read -u $fd -r headerline
    do
	[[ $headerline == $'\015' ]] && break
    done
    while IFS= read -u $fd -r -e; do :; done
    ztcp -c $fd
}

Re: pure zsh implementation of wget

[Stephane Bortzmeyer]

On Thu, Sep 22, 2011 at 02:06:40PM -0700,
 Guido van Steen <gvsteen@yahoo.com> wrote 
 a message of 11 lines which said:

> However, when I run this script, I do not receive any files. 

It works fine for me.

% echo $ZSH_VERSION
4.3.10