API

Public API Functions

This module contains the public API functions. The goal is to gain the benefits of asynchronous HTTP requests while dropping one of these functions into a synchronous code flow. For basic usage, no prior knowledge of asynchronous programming or asyncio is required. Simply import aiodownload and call the functions with some URLs.

aiodownload.api.one(url_or_bundle, download=None)

Make one HTTP request and download it

Parameters:
  • url_or_bundle (str or AioDownloadBundle) – a URL string or bundle
  • download (AioDownload) – (optional) your own customized download object
Returns:

a bundle
Return type:

AioDownloadBundle
aiodownload.api.swarm(iterable, download=None)

Make a swarm of requests and download them

Parameters:
  • iterable (iterable object) – an iterable object (ex. list of URL strings)
  • download (AioDownload) – (optional) your own customized download object
Returns:

a list of bundles
Return type:

list
aiodownload.api.each(iterable, url_map=None, download=None)

For each iterable object, map it to a URL and request asynchronously

Parameters:
  • iterable (iterable object) – an iterable object (ex. list of objects)
  • url_map (callable object) – (optional) callable object mapping an object to a url or bundle
  • download (AioDownload) – (optional) your own customized download object
Returns:

generator

aiodownload API

class aiodownload.AioDownload(client=None, download_strategy=None, request_strategy=None)

The core class responsible for the coordination of requests and downloads

Parameters:
  • client (aiohttp.ClientSession) – (optional) client session, a default is instantiated if not provided
  • download_strategy (aiodownload.DownloadStrategy) – (optional) download strategy, a default is instantiated if not provided
  • request_strategy (aiodownload.RequestStrategy) – (optional) request strategy, a Lenient strategy is instantiated if not provided
main(bundle)

Main entry point for task creation with an asyncio event loop.

The number of concurrent requests is throttled using this async method. Depending on the download strategy used, the method will call the request_and_download async method or immediately return the bundle indicating that the file came from cache as the file existed.

Parameters:bundle (aiodownload.AioDownloadBundle) – bundle (generally one that has just been instantiated)
Returns:bundle with updated properties reflecting it’s final state
Rtype bundle:aiodownload.AioDownloadBundle
request_and_download(bundle)

Make an HTTP request and write it to disk. Use the download and request strategies of the instance to implement how this is achieved.

Parameters:bundle (aiodownload.AioDownloadBundle) – bundle with it’s url and file_path set
Returns:bundle with updated properties reflecting success or failure
Rtype bundle:aiodownload.AioDownloadBundle

Strategies

This module contains base class and sensible default class implementations for the request and download strategies used by AioDownload

class aiodownload.strategy.DownloadStrategy(chunk_size=65536, home=None, skip_cached=False)

DownloadStrategy is an injection class for AioDownload. The purpose is to control download options for AioDownload.

Parameters:
  • chunk_size ((optional) int) – the incremental chunk size to read from the response
  • home – the base file path to use for writing response content to file
  • skip_cached (bool) – indicates whether existing written files should be skipped
get_file_path(bundle)

Get the file path for the bundle

Parameters:bundle (AioDownloadBundle) – bundle (generally, it’s file_path should be None)
Returns:full file_path for the bundle (via the default URL transformation)
Return type:str
on_fail(bundle)

Write an empty file

Parameters:bundle (AioDownloadBundle) – bundle (file_path should exist)
Returns:None
on_success(response, bundle)

Write the response to the file path indicated in the bundle

Parameters:
  • response (aiohttp.ClientResponse) – successful response from an HTTP response
  • bundle (AioDownloadBundle) – bundle (file_path should exist)
Returns:

None
class aiodownload.strategy.RequestStrategy(concurrent=2, max_attempts=0, timeout=60)

RequestStrategy is an injection class for AioDownload. The purpose is to control how AioDownload performs requests and retries requests.

Parameters:
  • concurrent ((optional) int) – the number of concurrent asynchronous HTTP requests to maintain
  • max_attempts (int) – maximum number of attempts before giving up
  • time_out (int) – timeout for the client session
assert_response(response)

Assertion for the response

Parameters:response (aiohttp.ClientResponse) – response from an HTTP response
Returns:None or AssertionError
get_sleep_time(bundle)

Returns how much time the bundle should sleep based on it’s properties

Parameters:bundle (AioDownloadBundle) – bundle
Returns:sleep time
Return type:int
retry(response)

Retry the HTTP request based on the response

Parameters:response (aiohttp.ClientResponse) – response from an HTTP response
Returns:
Return type:bool
class aiodownload.strategy.Lenient(max_attempts=5)

Lenient request strategy designed for an average web server. Try five times with a minute between each retry.

get_sleep_time(bundle)

Returns how much time the bundle should sleep based on it’s properties

Parameters:bundle (AioDownloadBundle) – bundle
Returns:sleep time
Return type:int
retry(response)

Retry any unsuccessful HTTP response except a 404 (if they say it’s not there, let’s believe them)

class aiodownload.strategy.BackOff(max_attempts=10)

Back Off request strategy designed for APIs and web servers that can be hammered a little harder. Exponentially back away if a failed response is returned.

get_sleep_time(bundle)

Returns how much time the bundle should sleep based on it’s properties

Parameters:bundle (AioDownloadBundle) – bundle
Returns:sleep time
Return type:int