fastdfs

FastDFS is an open-source, high-performance distributed file system (DFS) designed to handle file storing, syncing, and accessing (uploading and downloading). It addresses high capacity and load balancing challenges, making it suitable for file-based services such as photo sharing and video sharing sites. Key features include: - Grouped storage servers with a simple and flexible peer-to-peer structure, eliminating single points of failure - File IDs generated by FastDFS serve as credentials for file access, removing the need for traditional name servers or meta servers - Support for large, medium, and small files, including merged storage for small files to handle massive numbers of small files - Storage servers support multiple disks with data recovery for single disk failures - An nginx extension module for seamless integration with nginx - Support for IPv6, NAT networks, and cross-data-center or hybrid cloud deployment - Read-write separation and cross-data-center disaster backup - APIs for truncate, append, and modify operations on appender files - Multi-threaded file upload and download with resume from broken point support Architecture: FastDFS operates with two roles: tracker and storage. The tracker handles scheduling and load balancing for file access, while storage servers store files and manage file metadata (key-value pairs like width=1024). Both tracker and storage clusters can have servers added or removed without affecting online services. Storage servers are organized into file groups for high capacity, with servers within a group backing each other up and load balancing. When storage capacity is insufficient, additional groups can be added to expand capacity. File identification consists of two parts: the group name and the file name. Client SDKs are available for: - C Language (included in the source code) - PHP extension (included in the source code) - Java SDK (separate repository) - Go SDK (separate repository) FastDFS is a lightweight object storage solution. For general distributed file systems requiring strong data consistency for databases, K8s, and virtual machines, FastCFS is recommended as an alternative.