API¶
Public API Functions
This module contains the public API functions. The goal is to gain the benefits of asynchronous HTTP requests while dropping one of these functions into a synchronous code flow. For basic usage, no prior knowledge of asynchronous programming or asyncio is required. Simply import aiodownload and call the functions with some URLs.
aiodownload.api.
one
(url_or_bundle, download=None)¶Make one HTTP request and download it
Parameters:
- url_or_bundle (str or
AioDownloadBundle
) – a URL string or bundle- download (
AioDownload
) – (optional) your own customized download objectReturns: a bundle
Return type:
AioDownloadBundle
aiodownload.api.
swarm
(iterable, download=None)¶Make a swarm of requests and download them
Parameters:
- iterable (iterable object) – an iterable object (ex. list of URL strings)
- download (
AioDownload
) – (optional) your own customized download objectReturns: a list of bundles
Return type: list
aiodownload.api.
each
(iterable, url_map=None, download=None)¶For each iterable object, map it to a URL and request asynchronously
Parameters:
- iterable (iterable object) – an iterable object (ex. list of objects)
- url_map (callable object) – (optional) callable object mapping an object to a url or bundle
- download (
AioDownload
) – (optional) your own customized download objectReturns: generator
aiodownload API
- class
aiodownload.
AioDownload
(client=None, download_strategy=None, request_strategy=None)¶The core class responsible for the coordination of requests and downloads
Parameters:
- client (
aiohttp.ClientSession
) – (optional) client session, a default is instantiated if not provided- download_strategy (
aiodownload.DownloadStrategy
) – (optional) download strategy, a default is instantiated if not provided- request_strategy (
aiodownload.RequestStrategy
) – (optional) request strategy, aLenient
strategy is instantiated if not provided
main
(bundle)¶Main entry point for task creation with an asyncio event loop.
The number of concurrent requests is throttled using this async method. Depending on the download strategy used, the method will call the request_and_download async method or immediately return the bundle indicating that the file came from cache as the file existed.
Parameters: bundle ( aiodownload.AioDownloadBundle
) – bundle (generally one that has just been instantiated)Returns: bundle with updated properties reflecting it’s final state Rtype bundle: aiodownload.AioDownloadBundle
request_and_download
(bundle)¶Make an HTTP request and write it to disk. Use the download and request strategies of the instance to implement how this is achieved.
Parameters: bundle ( aiodownload.AioDownloadBundle
) – bundle with it’s url and file_path setReturns: bundle with updated properties reflecting success or failure Rtype bundle: aiodownload.AioDownloadBundle
Strategies
This module contains base class and sensible default class implementations
for the request and download strategies used by AioDownload
- class
aiodownload.strategy.
DownloadStrategy
(chunk_size=65536, home=None, skip_cached=False)¶DownloadStrategy is an injection class for AioDownload. The purpose is to control download options for AioDownload.
Parameters:
- chunk_size ((optional) int) – the incremental chunk size to read from the response
- home – the base file path to use for writing response content to file
- skip_cached (bool) – indicates whether existing written files should be skipped
get_file_path
(bundle)¶Get the file path for the bundle
Parameters: bundle ( AioDownloadBundle
) – bundle (generally, it’s file_path should be None)Returns: full file_path for the bundle (via the default URL transformation) Return type: str
on_fail
(bundle)¶Write an empty file
Parameters: bundle ( AioDownloadBundle
) – bundle (file_path should exist)Returns: None
on_success
(response, bundle)¶Write the response to the file path indicated in the bundle
Parameters:
- response (
aiohttp.ClientResponse
) – successful response from an HTTP response- bundle (
AioDownloadBundle
) – bundle (file_path should exist)Returns: None
- class
aiodownload.strategy.
RequestStrategy
(concurrent=2, max_attempts=0, timeout=60)¶RequestStrategy is an injection class for AioDownload. The purpose is to control how AioDownload performs requests and retries requests.
Parameters:
- concurrent ((optional) int) – the number of concurrent asynchronous HTTP requests to maintain
- max_attempts (int) – maximum number of attempts before giving up
- time_out (int) – timeout for the client session
assert_response
(response)¶Assertion for the response
Parameters: response ( aiohttp.ClientResponse
) – response from an HTTP responseReturns: None or AssertionError
get_sleep_time
(bundle)¶Returns how much time the bundle should sleep based on it’s properties
Parameters: bundle ( AioDownloadBundle
) – bundleReturns: sleep time Return type: int
retry
(response)¶Retry the HTTP request based on the response
Parameters: response ( aiohttp.ClientResponse
) – response from an HTTP responseReturns: Return type: bool
- class
aiodownload.strategy.
Lenient
(max_attempts=5)¶Lenient request strategy designed for an average web server. Try five times with a minute between each retry.
get_sleep_time
(bundle)¶Returns how much time the bundle should sleep based on it’s properties
Parameters: bundle ( AioDownloadBundle
) – bundleReturns: sleep time Return type: int
retry
(response)¶Retry any unsuccessful HTTP response except a 404 (if they say it’s not there, let’s believe them)
- class
aiodownload.strategy.
BackOff
(max_attempts=10)¶Back Off request strategy designed for APIs and web servers that can be hammered a little harder. Exponentially back away if a failed response is returned.
get_sleep_time
(bundle)¶Returns how much time the bundle should sleep based on it’s properties
Parameters: bundle ( AioDownloadBundle
) – bundleReturns: sleep time Return type: int