Ceph is an open-source software-defined distributed file system. Ceph allows you to create a fault-tolerant distributed data storage available over the TCP/IP protocol. Ceph offers several storage access interfaces: object, block, and file. In this article, we’ll show how to install and configure a ceph cluster using the Quincy release (17) in this example.
In the minimum configuration, it is recommended to use three hosts (nodes) for a ceph cluster with 2 CPUs and 4 GB of RAM. In my example, I’m using Rocky Linux 8.
The ceph cluster storage system consists of several daemons:
- MON (Ceph monitor) – cluster monitor that tracks its status. All nodes in the cluster report information about their state to the monitors;
- OSD (Object Storage Device) – cluster element that stores data and processes client requests. Data in the OSD is stored in blocks;
- MDS (Metadata Server Daemon) – metadata server. Used to operate the CephFS file system (not used in this example);
- MGR (Manager Daemon) – monitoring service.
You must first prepare all the cluster nodes:
- Update operating system and packages to the latest versions
- Configure time synchronization using Chrony
- Create DNS entries for hosts. In hostname, you need to use the short name, not the FQDN.
Then you need to install Python3 and Podman on all servers (Ceph services run in containers). It is OK to use Docker instead of Podman:
# dnf install python39 # python3 --version
# dnf install podman # podman -v
podman version 4.0.2
Install the cephadm tool, which we will use to create and configure the cluster:
# curl --silent --remote-name --location https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm # chmod +x cephadm # ./cephadm add-repo --release quincy
Writing repo to /etc/yum.repos.d/ceph.repo...
Completed adding repo.
# ./cephadm install
Installing packages ['cephadm']...
To create a ceph cluster, cephadm bootstrap is used. This command will start the ceph services and the first Monitor on the specified node, create a cluster, generate keys, configuration file, etc.
In a production environment, it is recommended to use a separate network for replication traffic between OSDs. Therefore, you must first configure the network interfaces on the hosts.
# cephadm bootstrap --mon-ip 220.127.116.11 --cluster-network 192.168.1.0/24
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 4.0.2 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: GUID
Adding host ceph01...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Ceph Dashboard will also be launched:
Ceph Dashboard is now available at:
List running containers on the host:
# podman ps
Check cluster status:
# cephadm install ceph-common # ceph -s
In this example, we use the first node for management tasks, so you need to install SSH keys and place them in /etc/ceph on the rest nodes:
# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-02 # ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-03
Add hosts to the ceph cluster:
# ceph orch host add ceph-02 # ceph orch host add ceph-03
# ceph orch host ls
After some time, Ceph will start the Monitor and Mgr services on certain hosts according to its policy.
You can force the hosts where you want to run the ceph monitor:
# ceph orch apply mon --placement="2 ceph-01 ceph-02"
Check cluster status and active roles:
# ceph -s
Now you can start the OSD services and specify the disks that can be used for ceph storage:
# ceph orch daemon add osd ceph-01:/dev/sdb,/dev/sdc,/dev/sdd # ceph orch daemon add osd ceph-02:/dev/sdb,/dev/sdc,/dev/sdd # ceph orch daemon add osd ceph-03:/dev/sdb,/dev/sdc,/dev/sdd
In a productive environment, it is recommended to move DB and WAL (Write-ahead log) to SSD. Use the command:
# ceph orch daemon add osd host:data_devices=/dev/sdb,/dev/sdc,db_devices=/dev/sdd,osds_per_device=2
In this example, data_devices is a regular HDD and db_devices is an SSD device.
Check OSD status:
# ceph osd tree
A separate OSD daemon is created for each disk.
1 stray daemon(s) not managed by cephadm
Restart the ceph-mgr service on this node:
# ceph orch daemon restart mgr.ceph-01.dseows
This completes the ceph cluster setup, and you can connect your clients using the following protocols:
- Block access (RBD, Rados Block Device);
- File (CephFS);
- Object – S3 (RadosGW).