bulk_download

Contents

bulk_download#

stable_pretraining.data.bulk_download(urls: Iterable[str], dest_folder: str | Path, backend: str = 'filesystem', cache_dir: str = '~/.stable_pretraining/')[source]#

Download multiple files concurrently.

Example:

import stable_pretraining

stable_pretraining.data.bulk_download(
    [
        "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz",
        "https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz",
    ],
    "todelete",
)
Parameters:

  • urls (Iterable[str]) – List of URLs to download

  • dest_folder (Union[str, Path]) – Destination folder for downloads

  • backend (str, optional) – Storage backend type. Defaults to “filesystem”.

  • cache_dir (str, optional) – Cache directory path. Defaults to “~/.stable_pretraining/”.

Contents