sh 用wget下载整个页面

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了sh 用wget下载整个页面相关的知识,希望对你有一定的参考价值。

Wget is a command-line utility that can retrieve all kinds of files over the HTTP and FTP protocols. Since websites are served through HTTP and most web media files are accessible through HTTP or FTP, this makes Wget an excellent tool for ripping websites.

While Wget is typically used to download single files, it can be used to recursively download all pages and files that are found through an initial page:

wget -r -p //www.makeuseof.com
However, some sites may detect and prevent what you’re trying to do because ripping a website can cost them a lot of bandwidth. To get around this, you can disguise yourself as a web browser with a user agent string:

wget -r -p -U Mozilla //www.makeuseof.com
If you want to be polite, you should also limit your download speed (so you don’t hog the web server’s bandwidth) and pause between each download (so you don’t overwhelm the web server with too many requests):

wget -r -p -U Mozilla --wait=10 --limit-rate=35K //www.makeuseof.com
Wget comes bundled with most Unix-based systems. On Mac, you can install Wget using a single Homebrew command: brew install wget (how to set up Homebrew on Mac). On Windows, you’ll need to use this ported version instead.

以上是关于sh 用wget下载整个页面的主要内容,如果未能解决你的问题,请参考以下文章

wget 递归下载整个网站(网站扒皮必备)

linux扒站命令之利用wget快速扒站利用wget快速扒站

wget 下载整个网站,或者特定目录

wget 下载整个网站,或者特定目录

Centos下wget下载整个网站,或者目录全部文件

wget命令怎么用