Common home-storage choices occupy different product and technology layers.

The capacity bars show aggregate proportions, with metadata and recovery reserves excluded. They do not represent physical partitions. Different replica colors distinguish capacity overhead, with no primary/secondary relationship implied.

Single-disk redundancy leaves no redundancy after one disk fails. Another failure during rebuilding can then cause data loss. For home storage that runs for years while disks age, that is a significant reliability concern. I therefore use configurations able to preserve data through at least two disk failures as the baseline.

The configurations are RAID 6, Synology SHR-2, ZFS RAIDZ2, Unraid dual parity, and Ceph with three replicas.

For ordinary home storage such as photos, media, file sharing, and backups, a single-host system is usually sufficient. It costs less in hardware, power, and maintenance and is simpler to manage than a multi-node cluster.

Ceph is an open-source distributed storage project originating in Sage Weil’s research. Inktank later commercialized it, and Red Hat acquired Inktank in 2014. In 2022, Red Hat’s Ceph team moved to IBM. Development continues through the community and multiple vendors.

Its foundation is RADOS distributed object storage. Objects map through placement groups (PGs) and CRUSH rules to OSDs, the storage processes managing disks, with replicas retained according to policy. RBD, CephFS, and RGW expose block, file, and object interfaces. The architecture documentation explains the layers. The diagram uses three replicas across hosts.

My choice came down to several requirements.

The redundancy overhead is acceptable to me. My household dataset is manageable, HDD storage remains relatively inexpensive, and even in 2026 I still have a stock of enterprise disks formerly used for Chia farming.

bcache sits beneath the block device used by an OSD, using NVMe as a cache for HDD capacity.

Cache hits read from NVMe. Misses read from HDD and populate the cache according to policy. Writeback first writes to NVMe and later destages to HDD. Writethrough waits for the HDD write.

The benefit depends on access patterns and hit rate. My NAS contains many infrequently accessed files. Reading cold data for the first time still requires the HDD, limiting the improvement for that workload.

In actual use, I encountered potential stability issues and also had to consider cache-device failure. In writeback mode, NVMe may hold data awaiting destaging, making it a critical part of the storage path. Sharing one cache device across several backing disks can affect multiple devices at once and complicate diagnosis and recovery.

Another approach keeps data on HDD while placing BlueStore’s block.db on NVMe. block.db holds RocksDB metadata, and block.wal holds the write-ahead log. With a separate DB device, WAL normally uses that device too. See the BlueStore configuration reference.

A single NVMe can contain separate DB/WAL areas for several HDD OSDs. Each OSD coordinates access, using metadata to locate HDD data blocks and managing data persistence alongside metadata and log updates. Metadata already cached in memory needs no NVMe read.

Moving metadata provides no file-content cache. Cold reads still incur HDD seek and media latency.

Both approaches increased stability concerns and maintenance complexity in my environment, with limited improvements for cold reads. I eventually moved away from them. My experience informed that personal choice without establishing a universal defect in either design.

I separated HDDs and NVMe into independent pools, using device classes and CRUSH rules to control placement. Workloads needing higher IOPS, throughput, and lower latency use NVMe directly. Capacity-oriented workloads use HDDs, with placement and migration arranged at the application side.

My storage network runs at 10GbE. Tests and long-term use show that the HDD pool meets my needs for NAS files, VM image backups, and capacity-oriented object storage. Small files, random access, concurrency, and recovery load still affect the experience, so these results do not imply that every workload saturates 10GbE.

Here is an early CephFS file read/write test.

The test ran in a CephFS mount with a roughly 2.46 GB file. It reported 854 MB/s for writing and 2.8 GB/s for reading. Cache effects are included, so those values cannot be equated with sustained HDD persistence or network throughput. Together with everyday use, the result supports the suitability of file access performance for my NAS needs.

In mid-2025, before the subsequent broad rise in SSD prices, I replaced a node with an all-flash U.2 NVMe configuration, continuing the move toward flash storage.

I initially debated adding a UPS, mainly because I worried about data loss after sudden power cuts. A UPS also brings trade-offs.

Eaton’s UPS battery documentation discusses differences in safety and lifetime.

My tests produced no data loss. Over the years, the rack lost power roughly three or four times, again with no loss of already-written data. Those experiences made me comfortable with power-loss recovery in my own setup.

My requirement is that writes still incomplete at the instant of power loss may fail, while data confirmed as durable should remain. VM and database behavior still depends on transaction commits, synchronization, and devices honoring flush requests.

The tests and incidents cover a limited set of failures. Ceph replication works alongside device power-loss protection, correct persistence semantics, UPS protection, and independent backups. Replicas in the same rack also share the rack’s power-failure exposure.

Cross-node replication keeps data available on other nodes through such local failures. An HBA or backplane can affect several disks simultaneously, so failure-domain planning needs to consider more than individual disks. Maintenance also requires checking the health of other nodes to avoid overlapping failures.

I separate the PVE and Ceph networks. PVE carries compute-side management and workload traffic. Ceph carries storage access, OSD replication, heartbeats, and recovery.

The separation is especially useful during recovery. Failed disks, returning nodes, and rebalancing generate extra transfers. Keeping that traffic apart reduces bandwidth contention and makes the storage and compute networks easier to observe individually.

My design has only those two networks. Client access and OSD-to-OSD traffic share the Ceph network, with no further split between public_network and cluster_network. I chose to keep home-network complexity manageable. See the network configuration documentation.

Actual isolation also depends on physical links. VLANs separate logical networks but still share bandwidth on a common uplink. Capacity planning needs to cover both routine access and recovery.

I divide pools by medium and data class and choose redundancy separately. Three considerations guide that arrangement.

Devices need free space for recovery and rebalancing. Dividing raw capacity by replica count alone overestimates usable space. Pools sharing OSDs also share disk capacity and I/O resources.

For a home cluster whose alerts can be monitored and acted on promptly, I favor noout. Temporary network problems, reboots, or connection issues can otherwise cause a recoverable OSD to be automatically marked out, trigger data migration, and require another rebalance when it returns.

noout prevents down OSDs from being automatically marked out after the timeout. It neither keeps them online nor blocks all recovery and movement. The OSD troubleshooting documentation covers setting and clearing it during maintenance.

For planned work, version support may allow narrowing the scope to affected OSDs or hosts. Retaining cluster-wide noout means actively taking responsibility for failures.

Replicas address device and node failures. Recovering accidental deletion, erroneous overwrites, or malicious deletion requires a separate backup strategy. I retain static backups of cold data and back up selected critical data to cloud object storage.

Independent cold-data copies allow recovery without depending entirely on the original Ceph cluster. Cloud object storage adds a separate location so a rack-level incident cannot affect every copy.

Cloud backups store selected important data in object storage. I separate routine backup credentials from administrative credentials and limit access to required buckets and prefixes. Daily writers should especially lack permission to delete historical versions or change versioning settings. Versioning retains earlier copies after accidental overwrites or deletions.

Versioning also needs permissions and retention policies. Backup completion, availability of historical versions, and actual file recovery all need periodic checks.