Ceph is an extremely powerful distributed storage system which offers redundancy out of the box over multiple nodes beyond just single node setup. It is highly scalable and resilient to be used in an enterprise environment. Check out how CERN has been using Ceph to quench their immense thirst of big data need.
In this guide, we are going to learn how to configure Ceph Object Gateway to serve S3 compatible interface. The examples used in this guide are for 4 nodes Ceph cluster on Debian.
Prerequisite
It is important to ensure the Ceph cluster is healthy and no data rebalancing is in progress. A healthy Ceph cluster should appear as following after typing # ceph -s command:
cluster: id: 3921019cb-adfs3-4347-owier90-sl23498fjds
health: HEALTH_OK
services: mon: 3 daemons, quorum cph-01,cph-02,cph-03,cph-042
mgr: cph-01(active), standbys: cph-02,cph-03,cph-04
mds: cephfs-01-1/1/1 up {0=cph-01=up:active}, 2 up:standby
osd: 40 osds: 40 up, 40 in
rgw: 1 daemon active
data: pools: 10 pools, 2384 pgs
objects: 1.63M objects, 6.17TiB
usage: 18.5TiB used, 54.3TiB / 72.8TiB avail
pgs: 2383 active+clean
io:
client: 24.6MiB/s rd, 2.23MiB/s wr, 271op/s rd, 126op/s wr
Also, ensure that all member nodes in the Ceph cluster are fully updated.
Creating Keyring File
In Ceph, a Keyring file stores Ceph authentication keys and their associated permissions rights specifications. Authentication is extremely important in Ceph as it protects against a man-in-the-middle attack. Important to keep in mind that, authentication data is sent in the network is not encrypted. This may include authentication keys, permission info etc. Ceph is designed to be used inside a fully trusted environment.
We are going to use ceph-authtool to create the required Keyring file:
$ ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
Creating Keys for RADOSGW
We are going to use the same ceph-authtool to generate the required keys and add them to the previously created keyring:
[email protected]:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.cph1 --gen-key
[email protected]:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.cph2 --gen-key
[email protected]:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.cph3 --gen-key
[email protected]:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.cph4 --gen-key
Adding Capabilities
In this step we are going to add read, write and execute capabilities to the previously created keys:
[email protected]:~# ceph-authtool -n client.radosgw.cph1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
[email protected]:~# ceph-authtool -n client.radosgw.cph2 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
[email protected]:~# ceph-authtool -n client.radosgw.cph3 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
[email protected]:~# ceph-authtool -n client.radosgw.cph4 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
Adding Keys to the Cluster
Once the keys are generated and capabilities are added, we can add the keys to the cluster:
[email protected]:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.cph1 -i /etc/ceph/ceph.client.radosgw.keyring
[email protected]:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.cph2 -i /etc/ceph/ceph.client.radosgw.keyring
[email protected]:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.cph3 -i /etc/ceph/ceph.client.radosgw.keyring
[email protected]:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.cph4 -i /etc/ceph/ceph.client.radosgw.keyring
Editing Ceph Configuration
Ceph configuration file is located in /etc/ceph/ceph.conf. Edit it to add the following lines:
[client.radosgw.cph1]
host = cph1
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.radosgw.$host.log
rgw_dns_name = s3.domain.com
[client.radosgw.cph2]
host = cph2
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.radosgw.$host.log
rgw_dns_name = s3.domain.com
[client.radosgw.cph3]
host = cph3
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.rados.$host.log
rgw_dns_name = s3.domain.com
[client.radosgw.cph4]
host = cph4
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.rados.$host.log
rgw_dns_name = s3.domain.com
Installing RADOSGW Daemon
Access each node in the cluster and install RADOSGW package:
[email protected]:~# apt install radosgw
[email protected]:~# systemctl restart radosgw
Test RADOSGW S3 Interface
At this stage, RADOSGW has created few default pools and it should be ready to be accessed. We can test if RADOSGW installation was successful by accessing any nodes on port 7480, https://<node_ip>:7480. If RADOSGW is fully configured we should see some XML code as following:
<ListAllMyBucketsResult xmlns="https://s3.amazonaws.com/doc/2006-03-01/">
<Owner>
<ID>anonymous</ID>
<DisplayName/>
</Owner>
<Buckets/>
</ListAllMyBucketsResult>
Enabling SSL using Civetweb
By default, SSL is not enabled in RADOSGW. SSL can be enabled using either Civetweb or FastCGI. In this guide, we are going to use Civetweb to enable SSL for Ceph Object Storage. Although SSL can be configured Self-signed SSL, it is highly recommended to use paid 3rd party SSL provider or Let’s Encrypt SSL. This is important if the S3 interface would need to be exposed to public access over the Internet. Check out the SSL offerings by Symmcom powered by Sectigo, formerly Comodo CA.
Preparing SSL Certificates
Enabling SSL in RADOSGW requires a single certificate file with the main certificate, CA certificate and private key added. For our example, we have three files:
- Main certificate file: cph_domain_com.crt
- CA bundle file: sectigo_ca_bundle.crt
- Private Key file: cph_domain_com.key
The CA bundle usually contains all necessary certificates provided by the Certificate Authority. We are going to combine all three files into a single file named cph_radosgw.pem as following:
[email protected]:~# cat /etc/ssl/certs/sectigo_ca_bundle.crt >> /etc/ssl/certs/cph_radosgw.pem
[email protected]:~# cat /etc/ssl/private/cph_domain_com.key >> /etc/ssl/certs/cph_radosgw.pem
Copy the certificate file to all member nodes in the cluster.
Configuring SSL
Add the following line in /etc/ceph/ceph.conf. Change IP as needed based on the network environment. Note that the ‘s’ after the port number is not typo:
rgw_frontends = civetweb port=10.0.0.11:443s num_threads=50 ssl_certificate=/etc/ssl/certs/cph_domain_com.pem
The final configuration for RADOSGW will be as following:
[client.radosgw.cph1]
host = cph1
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.radosgw.$host.log
rgw_dns_name = s3.domain.com
rgw_frontends = civetweb port=10.0.0.11:443s num_threads=50 ssl_certificate=/etc/ssl/certs/cph_domain_com.pem
Restart RADOSGW service using the following command to apply the changes:
[email protected]:~# systemctl restart radosgw
Troubleshooting
Although the installation process of RADOSGW is very straight forward, issues can occur due to a few common mistake and misconfiguration.
DNS Name
Misconfigured DNS name in the configuration file. The rgw_dns_name in the Ceph configuration file is how RADOSGW will respond any request to. So if the value entered as FQDN but you are trying to access it with IP address, the S3 interface will be inaccessible.
Wildcard Subdomain With Cloudflare
It is a common practice to allow users to connect over the Internet to S3 object storage using FQDN. Usually, the format is bucketname.s3.domain.com or something similar. If the configured RADOSGW is placed in a multi-tenant environment where different users all from different entities need to access their own S3 buckets, then using bucketname.s3.domain.com is a better way to go.
In such a scenario, each bucket name would require an A record in the nameserver associated with the domain name. Depending on the number of users, manual creation of these DNS records can become a tedious task. So a DNS record such as *.s3.domain.com will work best for all bucket users. The free plan of Cloudflare does not offer the creation of wildcard record for a subdomain. Create the records manually or use Cloudflare Enterprise Plan.
SSL Certificate File
SInce RADOSGW require a single certificate file, an error can occur when combining all the certificate files into one. Ensure to have the certificate content as following order:
- Main certificate file = domain_com.crt
- CA Bundle file = ca_bundle.crt
- Private Key = domain_com.key
Refer to section Configuring SSL for a full command to combine certificate files.
Basic RADOSGW Commands
Ceph Object Gateway daemon RADOSGW comes with a wide range of commands to manage every aspect of the storage. Visit RADOSGW Documentation for more detailed commands.
Command Format
RADOSGW includes a utility program for all administrative tasks named radosgw-admin. The command format for all user related management is:
$ radosgw-admin user [options …]
To create a new user:
$ radosgw-admin user create --display-name=”John Doe” --uid=jdoe --email=This email address is being protected from spambots. You need JavaScript enabled to view it.
To remove the user and purge all data:
$ radosgw-admin user rm --uid=jdoe --purge-data
To check full info of a specific user:
$ radosgw-admin user stats --uid=jdoe
The command format for all bucket related management is:
$ radosgw-admin bucket [options …]
To see a full list of buckets:
$ radosgw-admin bucket list
To link a user to a specific bucket:
$ radosgw-admin bucket link --bucket=<bucket_name> --uid=<user_name>
To unlink a user to a specific bucket:
$ radosgw-admin bucket unlink --bucket=<bucket_name> --uid=<user_name>
The command format for to manage RADOSGW Realm is::
$ radosgw-admin realm [options …]
