pillar-csi

pillar-csi is a Kubernetes CSI driver for self-hosted bare-metal clusters. It takes local ZFS zvols or LVM logical volumes on a dedicated storage node and exports them to the rest of the cluster over NVMe-oF/TCP, writing directly to the kernel via configfs rather than relying on SSH, Python daemons, or external target CLIs. It is explicitly not a distributed filesystem: it does not replicate, stripe, or pool storage across nodes. Architecture consists of three workloads. pillar-controller runs as a Deployment and reconciles cluster-scoped Pillar* CRDs. pillar-agent runs as a DaemonSet on storage nodes only and owns all configfs writes on the host. pillar-node runs on every worker and handles the CSI Node service (initiator connect, mkfs, bind-mount). Both DaemonSets use hostNetwork so the NVMe-oF/TCP data plane can bind to the host network namespace. Configuration is declarative through CRDs: PillarAgent locates a storage agent, PillarStore describes a storage pool (ZFS pool name, LVM VG, backend config), PillarProtocol describes network protocol configuration, and PillarStorageClass combines pool and protocol into an auto-generated StorageClass. An internal PillarVolumeState CRD records durable state for recovery from partial provisioning failures and publication tracking; the controller enforces CSI access-mode exclusivity from that record. A notable design element is stale-operation fencing. Every agent call that changes a volume's resources carries a fencing token composed of the volume state UID (lifecycle identity) and a publicationGeneration counter bumped via compare-and-swap. The agent keeps durable marks per volume on the storage node's local disk under /var/lib/pillar-csi/agent/generations/, mounted as a hostPath so marks survive restarts and reboots. Requests older than the mark, from an ended or replaced lifecycle, or carrying no token are rejected with FAILED_PRECONDITION. The README notes an upgrade caveat: detach all volumes before upgrading, since earlier versions recorded no publications, lifecycles, or generations, and no migration shim is provided. Node stage state is recorded under /var/lib/pillar-csi/node/ on the worker, written atomically via temp-file, sync, and rename. NodeUnstageVolume reads the record back because the CO sends neither volume capability nor volume context on unstage. Unmount decisions delegate to the mounter's idempotent Unmount, treating corrupted-mount probe errors (EIO, ENOTCONN, ESTALE, EACCES) as still-mounted so kubelet can reap pods whose filesystem entered kernel shutdown. Supported matrix: ZFS zvol and LVM LV over NVMe-oF/TCP are shipped; iSCSI is designed but not yet shipped; NFS for ZFS datasets is designed but not yet shipped. CSI operations include CreateVolume, DeleteVolume, ControllerPublish/Unpublish, ControllerExpandVolume, NodeStage/Unstage, NodePublish/Unpublish, NodeExpandVolume, NodeGetVolumeStats, ValidateVolumeCapabilities, and GetCapacity. Access modes are ReadWriteOnce, ReadWriteOncePod, and ReadOnlyMany; volume modes are Filesystem (ext4/xfs) and Block. Installation is via Helm. mTLS between controller and agent is opt-in (cert-manager mode or supplied Secret mode); the default is plaintext gRPC. Kubernetes 1.24 or newer is required. Storage nodes need nvmet and nvmet_tcp kernel modules; worker nodes need nvme_tcp and nvme_fabrics; init-containers run modprobe on startup. The README includes a quickstart with CRD YAML examples, troubleshooting via standard kubectl describe and logs, and a detailed recovery procedure for legacy volumes stuck at ExportSpecMissing, emphasizing explicit operator decisions for bindAddress, port, and aclEnabled rather than guessing from runtime observations.