HB Computer Security | Aide Memoire
  • Introduction
  • Core Technical Skills
    • Core Skills
      • Linux
        • Getting to Know Linux
          • Using the Shell
            • Shells, Terminals, and Virtual Consoles
            • Choose your Shell
            • Shell Commands
              • Non-PATH Commands
              • Command History
                • Command Line Editing
                  • Keystrokes for Navigating Command Lines
                  • Keystrokes for Editing Command Lines
                  • Keystrokes for Cutting and Pasting Text from within Command Lines
                • Command Line Recall
                  • Keystrokes for Command Line Recall
              • Connecting and Expanding Commands
                • Piping Between Commands
                • Sequential Commands
                • Expanding Commands
            • Shell Variables
              • Common Shell Variables
            • Aliases
            • Create your Own Shell Environment
              • Modification Ideas
          • Navigating the Linux File System (LFS)
            • Filesystem Commands
            • Listing Files and Directories
            • File Permissions and Ownership
              • Modifying Permissions with chmod
              • Modifying Default Permissions with umask
              • Change File Ownership with chown
            • Copying, Moving, and Removing Files
            • Finding Files
              • locate
              • find
              • grep
            • Downloading Files
              • axel
              • wget
              • curl
                • User-Agent: Googlebot
          • Working with Text Files
            • Using vim and vi to Edit Text Files
              • Starting with vi
              • Adding Text
              • Moving Around in the Text
            • Text Manipulation
        • System Administration
          • Installing Linux
            • Installing from Live Media
            • Installing in the Enterprise
            • Partitioning Hard Disks
              • Tips for Creating Partitions
          • Account Administration
            • The root Account
              • Becoming root with su
              • sudo
                • The /etc/sudoers File
                • Granting sudo privileges
                  • visudo Guidance
                • Useful sudo Hints
            • Other Administrative Accounts
            • Standard User Accounts
              • Risks of userdel: Orphaned Files
          • Graphical Remote Administration
            • Cockpit
              • Installation Guide
            • Remote Desktop Protocol with xrdp
              • Installation and Configuration
            • Remote Desktop with vnc
              • Installation and Configuration
              • Running VNC as a System Service
          • Managing Running Processes
            • Listing Processes
              • ps
              • top
              • htop
            • Backgrounding and Foregrounding
              • Starting a Background Process
              • Using Foreground and Background Commands
            • Killing and Recining Processes
              • kill and killall
          • Managing Software
            • Managing Software from the Desktop
            • Going Beyond the Limitations of Software Center
              • Debian Packages
                • Advanced Package Tool (apt)
                • Repositories
                • dpkg
        • Shell Scripting
          • Variables
            • Command Substitution
            • Arguments
          • Reading User Input
          • if, else, and elif
          • BOOLEAN Logic
          • Loops
            • for Loops
            • while Loops
          • Functions
          • Local Vs Global Variables
          • Summary
        • Securing Linux
      • Windows
        • Security Hardening
Powered by GitBook
On this page
  • Additional Options
  • Specifying an Output Directory
  • Download Multiple Files
  • Resume a Download
  • Download in the Background
  • Change the User-Agent String
  • Download a Local Copy of a Website
  • Skipping the SSL Certificate Check
  • Download to Standard Output (STDOUT)

Was this helpful?

  1. Core Technical Skills
  2. Core Skills
  3. Linux
  4. Getting to Know Linux
  5. Navigating the Linux File System (LFS)
  6. Downloading Files

wget

PreviousaxelNextcurl

Last updated 1 year ago

Was this helpful?

The wget command, which we will use extensively throughout this aide-memoire, downloads files using the HTTP/HTTPS and FTP protocols. The example below shows the use of wget along with the -O switch to save the destination file with a different name on the local machine:

wget -O report_wget.pdf https://www.offensive-security.com/reports/penetration-testing-sample-report-2013.pdf

Additional Options

wget has some other handy options that you should be familiar with.

Specifying an Output Directory

We can specify a directory to download a file to with the -P option:

wget -P /mnt/iso http://mirrors.mit.edu/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1804.iso

Download Multiple Files

If you want to download multiple files at once, use the -i option, followed by the path to a local or external file containing a list of the URLs to be downloaded. Each URL needs to be on a separate line.

The following example shows how to download the Arch Linux, Debian, and Fedora iso files using the URLs specified in the linux-distros.txt file:

wget -i linux-distros.txt
http://mirrors.edge.kernel.org/archlinux/iso/2018.06.01/archlinux-2018.06.01-x86_64.iso
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.iso
https://download.fedoraproject.org/pub/fedora/linux/releases/28/Server/x86_64/iso/Fedora-Server-dvd-x86_64-28-1.1.iso

Resume a Download

We can also resume a download that has dropped off with -c:

wget -c http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.iso

If the remote server does not support resuming downloads, wget will start the download from the beginning and overwrite the existing file.

Download in the Background

We can also run downloads in the background with the -b option:

wget -b https://download.opensuse.org/tumbleweed/iso/openSUSE-Tumbleweed-DVD-x86_64-Current.iso

By default, the output is redirected to wget-log file in the current directory. To watch the status of the download, use the tail command:

$ tail -f wget-log

Change the User-Agent String

We can also manually change the User-Agent string so wget can appear as any browser when downloading a file via the --user-agent= option:

wget --user-agent="Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" http://wget-forbidden.com/

Download a Local Copy of a Website

To create a mirror of a website with wget, use the -m option. This will create a complete local copy of the website by following and downloading all internal links as well as the website resources (JavaScript, CSS, Images):

wget -m https://example.com

If you want to use the downloaded website for local browsing, you will need to pass a few extra arguments to the command above:

wget -m -k -p https://example.com

The -k option will cause wget to convert the links in the downloaded documents to make them suitable for local viewing. The -p option will tell wget to download all necessary files for displaying the HTML page.

Skipping the SSL Certificate Check

If you want to download a file over HTTPS from a host that has an invalid SSL certificate, use the --no-check-certificate option:

wget --no-check-certificate https://domain-with-invalid-ss.com

Download to Standard Output (STDOUT)

In the following example, wget will quietly (-q) download and output the latest WordPress version to STDOUT(-O -) and pipe it to the tar utility, which will extract the archive to the /var/www directory.

wget -q -O - "http://wordpress.org/latest.tar.gz" | tar -xzf - -C /var/www

Downloading a file with wget, renaming the local copy