How to Install Ceph on Proxmox VE Dedicated Servers

A practical, end-to-end guide for building reliable, scalable, and production-ready storage

Ceph on Proxmox VE is often perceived as complex at first glance, but when implemented correctly, it becomes one of the most powerful storage solutions available for virtualized environments. For teams running dedicated servers, Ceph removes the dependency on external SANs while delivering high availability, scalability, and resilience.

This guide provides a clear, technical, documentation-style walkthrough – covering planning, hardware requirements, networking, installation, configuration, and operational best practices – so you can deploy Ceph confidently in real-world production environments.

1. What Is Ceph and Why Use It with Proxmox VE?

Ceph is a distributed storage platform designed to store data reliably across multiple nodes while automatically handling replication, recovery, and scaling.

Ceph provides three primary storage types:

• Block storage (RBD) for virtual machine disks
• Object storage (RGW) compatible with S3 and Swift APIs
• File storage (CephFS) for shared filesystem use cases

Proxmox VE integrates Ceph directly into its management layer, allowing administrators to create and manage shared storage without relying on NFS or iSCSI. This tight integration simplifies deployment while enabling advanced features such as live migration, HA, and automated recovery.

2. When Ceph Is the Right Choice

Ceph is most effective when deployed in environments that can fully leverage its distributed design.

Ceph is well suited when:

• At least three dedicated servers are available
• High availability for virtual machines is required
• Storage capacity is expected to grow over time
• Data protection at both disk and node level is needed

Ceph may not be suitable when:

• Only one or two servers are available
• Ultra-low latency is required without replication overhead
• Network bandwidth or disk resources are limited

Understanding these boundaries helps avoid unnecessary complexity.

3. Hardware Requirements

3.1 Node Specifications

Each Proxmox node must provide sufficient compute and memory resources to support both Ceph and virtual machine workloads.

A general guideline is to allocate approximately 1 GB of RAM per 1 TB of raw Ceph storage, in addition to memory required for the host OS and VMs.

3.2 Storage Disks

Ceph stores data using Object Storage Daemons (OSDs), each typically mapped to a single physical disk.

Key requirements:

• Disks must be presented as raw devices
• Hardware RAID must be disabled (HBA or IT mode)
• One OSD per physical disk is recommended

Common disk roles:

• HDDs for capacity-oriented pools
• SSDs for balanced performance
• NVMes for latency-sensitive workloads

Operating system disks should always be separate from Ceph storage disks, and disk types should not be mixed within the same pool.

3.3 Networking Requirements

Ceph is heavily dependent on network performance, especially during recovery and rebalancing operations.

Minimum configuration:

• Two 1 Gbps interfaces (testing only)

Recommended configuration:

• 10 Gbps or 25 Gbps networking
• Dedicated networks for different traffic types:
  • Public network for client and VM access
  • Cluster network for OSD replication and recovery

Proper network design is critical to maintaining consistent performance.

4. Network Design for Ceph on Proxmox

Separating Ceph traffic from VM traffic prevents storage operations from impacting guest workloads.

Example configuration:

This separation improves recovery speed, reduces congestion, and increases overall cluster stability under load.

5. Preparing Proxmox VE Nodes

Before installing Ceph, all Proxmox nodes must be correctly prepared to ensure a smooth deployment.

Each node should:

• Run the same Proxmox VE version
• Have correct hostname and DNS resolution
• Use time synchronization (Chrony or NTP)
• Allow passwordless SSH between nodes

Disk readiness must also be verified:

lsblk

OSD disks must be empty, unmounted, and free of partitions or filesystems.

6. Installing Ceph Using Proxmox VE

Step 1: Install Ceph Packages

From the Proxmox web interface:

• Navigate to Datacenter → Ceph → Install Ceph
• Select and install the same Ceph release on all nodes

This installs all required Ceph components.

Step 2: Create Monitor Nodes (MON)

Ceph monitors maintain cluster state and quorum.

• Go to Datacenter → Ceph → Monitor
• Create the first monitor
• Repeat until at least three MONs exist
  

An odd number of monitors ensures quorum reliability.

Step 3: Deploy Manager Daemons (MGR)

Ceph managers handle metrics, dashboards, and background services.

In modern Proxmox versions, managers are created automatically and require minimal configuration.

Step 4: Configure Ceph Networks

Define network ranges used by Ceph:

Public Network: 192.168.1.0/24

Cluster Network: 10.10.10.0/24

This ensures traffic is routed over the correct interfaces.

Step 5: Create OSDs

OSDs are created directly from the Proxmox interface:

• Navigate to Ceph → OSD → Create
• Select node and disk
• Optionally assign NVMe devices for BlueStore DB/WAL

Each disk becomes an independent storage unit within the cluster.

7. Creating Ceph Pools for Virtual Machines

Ceph pools define how data is stored and replicated.

Key parameters:

• Size: number of replicas (commonly 3)
• Min_size: minimum replicas required for I/O (commonly 2)

Typical pool layout:

Pools are added to Proxmox under Datacenter → Storage → RBD.

8. Performance Tuning and Optimization

Performance tuning ensures Ceph operates efficiently under load.

Key practices include:

• Using replication size appropriate for workload
• Assigning fast media (SSD/NVMe) to performance pools
• Using dedicated DB/WAL devices for BlueStore
• Avoiding overfilled pools

CRUSH rules should be used to separate disk classes so data is placed only on appropriate storage media.

9. Monitoring and Health Management

Ongoing monitoring is essential for cluster stability.

Available tools:

• Proxmox Ceph dashboard
• Command-line utilities such as:
  
  • ceph -s
  • ceph osd tree
  • ceph health detail

Administrators should regularly monitor disk usage, recovery operations, network latency, and cluster warnings.

10. Backup Strategy and Failure Handling

Ceph protects against hardware and node failures through replication, but it is not a backup solution.

Ceph handles:

• Disk failures
• Node outages
• Data consistency

Ceph does not replace:

• VM-level backups
• Off-site replication

Always combine Ceph with snapshot schedules and external backup systems such as Proxmox Backup Server.

11. Common Mistakes to Avoid

Frequent issues include:

• Using RAID instead of raw disks
• Deploying on insufficient network bandwidth
• Mixing disk types in the same pool
• Under-provisioning RAM
• Skipping network separation

Avoiding these pitfalls significantly improves long-term stability.

Conclusion

Deploying Ceph on Proxmox VE dedicated servers enables a self-healing, scalable, enterprise-grade storage platform without external storage dependencies. When designed with the right hardware, networking, and pool strategy, Ceph delivers consistent performance, resilience, and flexibility for modern virtualized workloads.

If you’re unsure how to size or design your Ceph cluster, you don’t need to figure it out alone – Netrouting engineers are happy to help you plan a setup that fits your workload and growth needs.