smart_open

smart_open is a Python 3 library that provides a unified, Pythonic API for streaming very large files to and from diverse storage systems including Amazon S3, Google Cloud Storage, Azure Blob Storage, HDFS, WebHDFS, HTTP/HTTPS, SFTP/SCP/SSH, and the local filesystem. It acts as a drop-in replacement for Python's built-in open() function, maintaining full compatibility while adding support for remote URIs and on-the-fly (de)compression for formats such as gzip, bzip2, LZ4, XZ, and Zstandard. The library handles authentication and transport details for each backend, allowing users to pass custom clients (e.g., boto3, google-cloud-storage, azure-storage-blob) via transport_params for advanced configuration. It supports context managers, seeking, and standard IOBase operations. Compression is inferred from file extensions by default but can be overridden or disabled. Additional utilities include parallel iteration over S3 bucket contents via smart_open.s3.iter_bucket(). Installation is minimal by default; optional dependencies for each storage backend and compression format can be installed via extras (e.g., pip install 'smart_open[s3,gcs,azure,http,webhdfs,ssh,zst,lz4]'). The project is well-tested, MIT-licensed, and actively maintained.