wget
Last updated
Last updated
The wget command, which we will use extensively throughout this aide-memoire, downloads files using the HTTP/HTTPS and FTP protocols. The example below shows the use of wget along with the -O switch to save the destination file with a different name on the local machine:
wget has some other handy options that you should be familiar with.
We can specify a directory to download a file to with the -P option:
If you want to download multiple files at once, use the -i option, followed by the path to a local or external file containing a list of the URLs to be downloaded. Each URL needs to be on a separate line.
The following example shows how to download the Arch Linux, Debian, and Fedora iso files using the URLs specified in the linux-distros.txt file:
We can also resume a download that has dropped off with -c:
If the remote server does not support resuming downloads, wget will start the download from the beginning and overwrite the existing file.
We can also run downloads in the background with the -b option:
By default, the output is redirected to wget-log file in the current directory. To watch the status of the download, use the tail command:
$ tail -f wget-log
We can also manually change the User-Agent string so wget can appear as any browser when downloading a file via the --user-agent= option:
To create a mirror of a website with wget, use the -m option. This will create a complete local copy of the website by following and downloading all internal links as well as the website resources (JavaScript, CSS, Images):
If you want to use the downloaded website for local browsing, you will need to pass a few extra arguments to the command above:
The -k option will cause wget to convert the links in the downloaded documents to make them suitable for local viewing. The -p option will tell wget to download all necessary files for displaying the HTML page.
If you want to download a file over HTTPS from a host that has an invalid SSL certificate, use the --no-check-certificate option:
In the following example, wget will quietly (-q) download and output the latest WordPress version to STDOUT(-O -) and pipe it to the tar utility, which will extract the archive to the /var/www directory.
