[TUHS] Mirror with link

[TUHS] Mirror with link

[Jose R. Valverde]

Just for the record.

The correct way to mirror a site with links corrected is

	wget -c -m -k -np -e robots=off URL

Seems most people have problem remembering this incantation.

-- 
			EMBnet/CNB
		Scientific Computing Service
	Solving all your computer needs for Scientific
			Research.

		http://bioportal.cnb.csic.es
		  http://www.es.embnet.org

[TUHS] Mirror with link

[Larry McVoy]

On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R. Valverde wrote:
> Just for the record.
> 
> The correct way to mirror a site with links corrected is
> 
> 	wget -c -m -k -np -e robots=off URL
> 
> Seems most people have problem remembering this incantation.

Wouldn't it be nice if it were

	wget --mirror URL

?
-- 
---
Larry McVoy                lm at bitmover.com           http://www.bitkeeper.com

[TUHS] Mirror with link

[Derrik Walker]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1115 bytes --]

I like shell scripts for hard to remember things like that.

Call it wget_mirror.sh or something like that:

#!/bin/bash

if [ $# -ne 1 ] ; then echo ERRR; exit 1 ; else url=$1 ; fi

wget -c -m -k -np -e robots=off $url


( that was off the top of my head and has NOT been tested ... it might have a typo or two :)

- Derrik

On Oct 13, 2011, at 02:43 PM, Larry McVoy <lm at bitmover.com> wrote:

> On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R. Valverde wrote:
> > Just for the record.
> >
> > The correct way to mirror a site with links corrected is
> >
> > wget -c -m -k -np -e robots=off URL
> >
> > Seems most people have problem remembering this incantation.
>
> Wouldn't it be nice if it were
>
> wget --mirror URL
>
> ?
> -- 
> ---
> Larry McVoy lm at bitmover.com http://www.bitkeeper.com
> _______________________________________________
> TUHS mailing list
> TUHS at minnie.tuhs.org
> https://minnie.tuhs.org/mailman/listinfo/tuhs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20111013/29ca20ef/attachment.html>

[TUHS] Mirror with link

[SPC]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

Computing woudln't be the same :-)

-- 
Saludos - Greetings - Freundliche Grüße - Salutations

Sergio Pedraja
<http://www.linkedin.com/in/sergiopedraja>
twitter: @sergio_pedraja
http://www.linkedin.com/in/sergiopedraja
 http://www.quora.com/Sergio-Pedraja
-----

2011/10/13 Larry McVoy <lm at bitmover.com>

> On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R. Valverde wrote:
> > Just for the record.
> >
> > The correct way to mirror a site with links corrected is
> >
> >       wget -c -m -k -np -e robots=off URL
> >
> > Seems most people have problem remembering this incantation.
>
> Wouldn't it be nice if it were
>
>        wget --mirror URL
>
> ?
> --
> ---
> Larry McVoy                lm at bitmover.com
> http://www.bitkeeper.com
> _______________________________________________
> TUHS mailing list
> TUHS at minnie.tuhs.org
> https://minnie.tuhs.org/mailman/listinfo/tuhs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20111013/9ccb1331/attachment.html>

[TUHS] Mirror with link

[A. P. Garcia]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

On Thu, Oct 13, 2011 at 1:43 PM, Larry McVoy <lm at bitmover.com> wrote:
> On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R. Valverde wrote:
>> Just for the record.
>>
>> The correct way to mirror a site with links corrected is
>>
>>       wget -c -m -k -np -e robots=off URL
>>
>> Seems most people have problem remembering this incantation.
>
> Wouldn't it be nice if it were
>
>        wget --mirror URL

for a complete mirror, yes. but if you want to make sure that you
don't download any malware:

wget --mirror --evil=false

[TUHS] Mirror with link

[Jose R. Valverde]

That is already included: -m == --mirror

But --mirror (or -m) does not include -k (convert links to local after
the transfer) nor -np (do not follow links upwards the parent directory),
nor an instruction to ignore 'robots.txt'.

The magic incantation I submitted will only download down the hierarchy,
in spite of robots.txt and fixing links, all three problems reported in
the thread.

Of course it is not polite to ignore robots.txt, but sometimes it may be
justified.

				j

On Thu, 13 Oct 2011 11:43:37 -0700
Larry McVoy <lm at bitmover.com> wrote:
> On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R. Valverde wrote:
> > Just for the record.
> > 
> > The correct way to mirror a site with links corrected is
> > 
> > 	wget -c -m -k -np -e robots=off URL
> > 
> > Seems most people have problem remembering this incantation.
> 
> Wouldn't it be nice if it were
> 
> 	wget --mirror URL
> 
> ?
> -- 
> ---
> Larry McVoy                lm at bitmover.com           http://www.bitkeeper.com


-- 
			EMBnet/CNB
		Scientific Computing Service
	Solving all your computer needs for Scientific
			Research.

		http://bioportal.cnb.csic.es
		  http://www.es.embnet.org