juicefs

JuiceFS is a high-performance POSIX file system released under Apache License 2.0 and designed for cloud-native environments. Files written through JuiceFS are persisted in object storage (for example Amazon S3 or other S3-compatible services), while metadata such as file names, sizes, permissions, timestamps and directory structure is kept in a separate metadata engine. Supported metadata engines include Redis, MySQL, SQLite and TiKV, so deployments can be tuned to different scenarios and requirements. The project describes three main parts: the JuiceFS client, which coordinates object storage and the metadata engine and implements file system interfaces; the data storage layer, which can use local disks, public or private cloud object storage, or HDFS; and the metadata engine. Files are split into chunks (default upper limit 64 MiB), chunks into slices, and slices into blocks (4 MiB by default) that are stored in object storage. Because of this layout, source files are not directly visible in an object storage browser; the bucket contains a chunks directory with numbered directories and files. Highlighted capabilities listed in the README include full POSIX compatibility, a Hadoop Java SDK compatible with Hadoop 2.x and 3.x, an S3-compatible gateway, a Kubernetes CSI driver, shared read/write access from thousands of clients, strong consistency where confirmed modifications are immediately visible to all mounted clients, low-latency and scalable throughput, encryption in transit and at rest, global file locks (BSD flock and POSIX fcntl), and data compression with LZ4 or Zstandard. The README also notes close-to-open consistency, atomic rename and metadata operations, accessible opened files after unlink, mmap, fallocate with punch hole support, and extended attributes. It states that JuiceFS passed all 8813 compatibility tests in the latest pjdfstest. Getting started requires a supported metadata engine, supported object storage, and the JuiceFS client. Documentation covers a quick start guide, command reference, containers (Docker and Podman), Kubernetes, Hadoop Java SDK, Redis best practices, object storage setup, cache, fault diagnosis, FUSE mount options, Windows usage and the S3 gateway. Benchmarks are provided through a JuiceFS subcommand, plus fio sequential read/write and mdtest metadata comparisons against EFS and S3FS; the README presents these as favorable results and links to details. Supported object storage includes Amazon S3 and S3-compatible services, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, Tencent Cloud COS, Qiniu Kodo, QingStor, Ceph RGW, MinIO, local disk and Redis, among others. The README says JuiceFS is production ready and used by thousands of machines, with an adopters list and integration documentation. The storage format is described as stable and supported by future releases. The roadmap mentions gateway optimization, resumable sync, read-ahead optimization, large-scale scenario optimization and snapshots. JuiceFS collects anonymous usage data by default, reporting only core metrics such as version number, and this can be disabled with the --no-usage-report option. The design was inspired by Google File System, HDFS and MooseFS. Community support is available through GitHub Discussions and Discord, and issues are tracked on GitHub.