From: jrvalverde@cnb.csic.es (Jose R. Valverde)
Subject: [TUHS] Mirror with link
Date: Fri, 14 Oct 2011 17:59:53 +0200 [thread overview]
Message-ID: <20111014175953.2b02bde2@cnb.csic.es> (raw)
In-Reply-To: <20111013184337.GA23950@bitmover.com>
That is already included: -m == --mirror
But --mirror (or -m) does not include -k (convert links to local after
the transfer) nor -np (do not follow links upwards the parent directory),
nor an instruction to ignore 'robots.txt'.
The magic incantation I submitted will only download down the hierarchy,
in spite of robots.txt and fixing links, all three problems reported in
the thread.
Of course it is not polite to ignore robots.txt, but sometimes it may be
justified.
j
On Thu, 13 Oct 2011 11:43:37 -0700
Larry McVoy <lm at bitmover.com> wrote:
> On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R. Valverde wrote:
> > Just for the record.
> >
> > The correct way to mirror a site with links corrected is
> >
> > wget -c -m -k -np -e robots=off URL
> >
> > Seems most people have problem remembering this incantation.
>
> Wouldn't it be nice if it were
>
> wget --mirror URL
>
> ?
> --
> ---
> Larry McVoy lm at bitmover.com http://www.bitkeeper.com
--
EMBnet/CNB
Scientific Computing Service
Solving all your computer needs for Scientific
Research.
http://bioportal.cnb.csic.es
http://www.es.embnet.org
prev parent reply other threads:[~2011-10-14 15:59 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-10-13 18:37 Jose R. Valverde
2011-10-13 18:43 ` Larry McVoy
2011-10-13 18:51 ` Derrik Walker
2011-10-13 18:53 ` SPC
2011-10-13 19:35 ` A. P. Garcia
2011-10-14 15:59 ` Jose R. Valverde [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20111014175953.2b02bde2@cnb.csic.es \
--to=jrvalverde@cnb.csic.es \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).