How To Configure S3 Compatible Object Storage on Ceph

How To Configure S3 Compatible Object Storage on Ceph

 Ceph is an extremely powerful distributed storage system which offers redundancy out of the box over multiple nodes beyond just single node setup. It is highly scalable and resilient to be used in an enterprise environment. Check out how CERN has been using Ceph to quench their immense thirst of big data need.

In this guide, we are going to learn how to configure Ceph Object Gateway to serve S3 compatible interface. The examples used in this guide are for 4 nodes Ceph cluster on Debian.


It is important to ensure the Ceph cluster is healthy and no data rebalancing is in progress. A healthy Ceph cluster should appear as following after typing # ceph -s command:

   id:    3921019cb-adfs3-4347-owier90-sl23498fjds
   health: HEALTH_OK
services:    mon: 3 daemons, quorum cph-01,cph-02,cph-03,cph-042
  mgr: cph-01(active), standbys: cph-02,cph-03,cph-04
   mds: cephfs-01-1/1/1 up  {0=cph-01=up:active}, 2 up:standby
   osd: 40 osds: 40 up, 40 in
   rgw: 1 daemon active
data:    pools:   10 pools, 2384 pgs
   objects: 1.63M objects, 6.17TiB
   usage:   18.5TiB used, 54.3TiB / 72.8TiB avail
   pgs:     2383 active+clean
   client:   24.6MiB/s rd, 2.23MiB/s wr, 271op/s rd, 126op/s wr

Also, ensure that all member nodes in the Ceph cluster are fully updated.

Creating Keyring File

In Ceph, a Keyring file stores Ceph authentication keys and their associated permissions rights specifications. Authentication is extremely important in Ceph as it protects against a man-in-the-middle attack. Important to keep in mind that, authentication data is sent in the network is not encrypted. This may include authentication keys, permission info etc. Ceph is designed to be used inside a fully trusted environment.

We are going to use ceph-authtool to create the required Keyring file:

$ ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring

Creating Keys for RADOSGW

We are going to use the same ceph-authtool to generate the required keys and add them to the previously created keyring:

root@cph1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.cph1 --gen-key
root@cph2:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.cph2 --gen-key
root@cph3:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.cph3 --gen-key
root@cph4:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.cph4 --gen-key

Adding Capabilities

In this step we are going to add read, write and execute capabilities to the previously created keys:

root@cph1:~# ceph-authtool -n client.radosgw.cph1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
root@cph2:~# ceph-authtool -n client.radosgw.cph2 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
root@cph3:~# ceph-authtool -n client.radosgw.cph3 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
root@cph4:~# ceph-authtool -n client.radosgw.cph4 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring

Adding Keys to the Cluster

Once the keys are generated and capabilities are added, we can add the keys to the cluster:

root@cph1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.cph1 -i /etc/ceph/ceph.client.radosgw.keyring
root@cph2:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.cph2 -i /etc/ceph/ceph.client.radosgw.keyring
root@cph3:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.cph3 -i /etc/ceph/ceph.client.radosgw.keyring
root@cph4:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.cph4 -i /etc/ceph/ceph.client.radosgw.keyring

Editing Ceph Configuration

Ceph configuration file is located in /etc/ceph/ceph.conf. Edit it to add the following lines:

       host = cph1
       keyring = /etc/pve/priv/ceph.client.radosgw.keyring
       log file = /var/log/ceph/client.radosgw.$host.log
       rgw_dns_name =
       host = cph2
       keyring = /etc/pve/priv/ceph.client.radosgw.keyring
       log file = /var/log/ceph/client.radosgw.$host.log
       rgw_dns_name =

       host = cph3
       keyring = /etc/pve/priv/ceph.client.radosgw.keyring
       log file = /var/log/ceph/client.rados.$host.log
       rgw_dns_name =

       host = cph4
       keyring = /etc/pve/priv/ceph.client.radosgw.keyring
       log file = /var/log/ceph/client.rados.$host.log
       rgw_dns_name =

Installing RADOSGW Daemon

Access each node in the cluster and install RADOSGW package:

root@cph1:~# apt install radosgw
root@cph1:~# systemctl restart radosgw

Test RADOSGW S3 Interface

At this stage, RADOSGW has created few default pools and it should be ready to be accessed. We can test if RADOSGW installation was successful by accessing any nodes on port 7480, https://<node_ip>:7480. If RADOSGW is fully configured we should see some XML code as following:

<ListAllMyBucketsResult xmlns="">

Enabling SSL using Civetweb

By default, SSL is not enabled in RADOSGW. SSL can be enabled using either Civetweb or FastCGI. In this guide, we are going to use Civetweb to enable SSL for Ceph Object Storage. Although SSL can be configured Self-signed SSL, it is highly recommended to use paid 3rd party SSL provider or Let’s Encrypt SSL. This is important if the S3 interface would need to be exposed to public access over the Internet. Check out the SSL offerings by Symmcom powered by Sectigo, formerly Comodo CA.

Preparing SSL Certificates

Enabling SSL in RADOSGW requires a single certificate file with the main certificate, CA certificate and private key added. For our example, we have three files:

  • Main certificate file: cph_domain_com.crt
  • CA bundle file: sectigo_ca_bundle.crt
  • Private Key file: cph_domain_com.key

The CA bundle usually contains all necessary certificates provided by the Certificate Authority. We are going to combine all three files into a single file named cph_radosgw.pem as following:

root@cph1:~# cat /etc/ssl/certs/sectigo_ca_bundle.crt >> /etc/ssl/certs/cph_radosgw.pem
root@cph1:~# cat /etc/ssl/private/cph_domain_com.key >> /etc/ssl/certs/cph_radosgw.pem

Copy the certificate file to all member nodes in the cluster.

Configuring SSL

Add the following line in /etc/ceph/ceph.conf. Change IP as needed based on the network environment. Note that the ‘s’ after the port number is not typo:

rgw_frontends = civetweb port= num_threads=50 ssl_certificate=/etc/ssl/certs/cph_domain_com.pem

The final configuration for RADOSGW will be as following:

       host = cph1
       keyring = /etc/pve/priv/ceph.client.radosgw.keyring
       log file = /var/log/ceph/client.radosgw.$host.log
       rgw_dns_name =
       rgw_frontends = civetweb port= num_threads=50 ssl_certificate=/etc/ssl/certs/cph_domain_com.pem

Restart RADOSGW service using the following command to apply the changes:

root@cph1:~# systemctl restart radosgw


Although the installation process of RADOSGW is very straight forward, issues can occur due to a few common mistake and misconfiguration.

DNS Name

Misconfigured DNS name in the configuration file. The rgw_dns_name in the Ceph configuration file is how RADOSGW will respond any request to. So if the value entered as FQDN but you are trying to access it with IP address, the S3 interface will be inaccessible.

Wildcard Subdomain With Cloudflare

It is a common practice to allow users to connect over the Internet to S3 object storage using FQDN. Usually, the format is or something similar. If the configured RADOSGW is placed in a multi-tenant environment where different users all from different entities need to access their own S3 buckets, then using is a better way to go.

In such a scenario, each bucket name would require an A record in the nameserver associated with the domain name. Depending on the number of users, manual creation of these DNS records can become a tedious task. So a DNS record such as * will work best for all bucket users. The free plan of Cloudflare does not offer the creation of wildcard record for a subdomain. Create the records manually or use Cloudflare Enterprise Plan.

SSL Certificate File

SInce RADOSGW require a single certificate file, an error can occur when combining all the certificate files into one. Ensure to have the certificate content as following order:

  • Main certificate file = domain_com.crt
  • CA Bundle file = ca_bundle.crt
  • Private Key = domain_com.key

Refer to section Configuring SSL for a full command to combine certificate files.

Basic RADOSGW Commands

Ceph Object Gateway daemon RADOSGW comes with a wide range of commands to manage every aspect of the storage. Visit RADOSGW Documentation for more detailed commands.

Command Format

RADOSGW includes a utility program for all administrative tasks named radosgw-admin. The command format for all user related management is:

$ radosgw-admin user [options …]

To create a new user:

$ radosgw-admin user create --display-name=”John Doe” --uid=jdoe --email=This email address is being protected from spambots. You need JavaScript enabled to view it.

To remove the user and purge all data:

$ radosgw-admin user rm --uid=jdoe --purge-data

To check full info of a specific user:

$ radosgw-admin user stats --uid=jdoe

The command format for all bucket related management is:

$ radosgw-admin bucket [options …]

To see a full list of buckets:

$ radosgw-admin bucket list

To link a user to a specific bucket:

$ radosgw-admin bucket link --bucket=<bucket_name> --uid=<user_name>

To unlink a user to a specific bucket:

$ radosgw-admin bucket unlink --bucket=<bucket_name> --uid=<user_name>

The command format for to manage RADOSGW Realm is::

$ radosgw-admin realm [options …]

Latest HOW-TOs