Batch Downloads from the IACDC

Downloading in batch from the IACDC is done using wget. To download in batch from the IACDC please follow the instructions below:

To download all files in a directory:

  1. In the download section of the relevant dataset, go to the directory containing the files you wish to download.
  2. Download the text file at the top of the directory called 1wget_8756 (note the numbers at the end change in each directory).
  3. From your command line execute the following command:

wget –cN -i 1wget_8756

  1. The download of all files in that directory will follow.

Should you not require all the files in the directory:

  1. In the download section of the relevant dataset, go to the directory containing the files you wish to download.
  2. Download the text file at the top of the directory called 1wget_8756 (note the numbers at the end change in each directory).
  3. Manually edit this file to contain only the download links you wish you download, i.e. delete the links for files that are not of interest.
  4. From your command line execute the following command:

wget –cN -i 1wget_8756

  1. The download of the relevant files in that directory will follow.

wget options:

Downloading using wget provides a large number of options depending on your needs. The following options we have found most useful when downloading data from the IACDC:

  • -c’ or ‘–continue’: Continue getting a partially-downloaded file. This is useful when you want to finish a previously started download.
  • -N’ or ‘–timestamping’: Turns on time-stamping.
  • -i file’ or ‘–input-file=file’: Reads URLs from a file. In this case the file is the 1wget_####.txt file found for each directory.

wget documentation, providing a full list of options and details can be found at https://www.gnu.org/software/wget/manual/