In the previous post we saw the deployment of OpenStack Platform 10 with Ceph.
Lets log into the Controller Node, and look at all the services running.
Lets now look at all the Ceph related packages installed on the Controller Node.
Lets verify if the required Ceph configurations are created to integrate with Glance. We will see that a Ceph user "client.openstack" with read permission for monitor, and read, write, execute permission for the created pool.
Lets log into the Controller Node, and look at all the services running.
[root@overcloud-controller-0 ~]# openstack service list
+----------------------------------+------------+----------------+
| ID | Name | Type |
+----------------------------------+------------+----------------+
| 316642206a194dc58cb4528adbc90f5d | heat-cfn | cloudformation |
| 593f9fde36634401977d2806a9e6f6f5 | gnocchi | metric |
| 6fd62024660942ec8d6451429f03d336 | ceilometer | metering |
| 762a53b0efd54947b3d6d2d6405440b0 | cinderv2 | volumev2 |
| 7783685d418845b9acc1bad50fbbfe7d | nova | compute |
| 93987d0d0fea4f02a73e7825c9b4adfe | glance | image |
| 9a9990c7780945aca10fbeba19cc4729 | aodh | alarming |
| a13c6596df864b17a33dc674a278c31c | cinderv3 | volumev3 |
| bdb018992e1146b499d5b56d8fdfce29 | heat | orchestration |
| d0e8d3c58784485888d5b36d71af39a4 | keystone | identity |
| d7a05ca55d2349d2b1f4dbc5fb79fb2d | cinder | volume |
| e238bbf05e8547e7b4b57ee4dba52a63 | neutron | network |
| fb32e87e977742899426a848077b09db | swift | object-store |
+----------------------------------+------------+----------------+
Lets now look at all the Ceph related packages installed on the Controller Node.
[root@overcloud-controller-0 ~]# yum list installed | grep -i ceph
ceph-base.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
ceph-common.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
ceph-mon.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
ceph-osd.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-osd-signed
ceph-radosgw.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-tools-signed
ceph-selinux.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
fcgi.x86_64 2.4.0-27.el7cp @rhos-10.0-ceph-2.0-mon-signed
gperftools-libs.x86_64 2.4-8.el7 @rhos-10.0-ceph-2.0-mon-signed
leveldb.x86_64 1.12.0-7.el7cp @rhos-10.0-ceph-2.0-mon-signed
libbabeltrace.x86_64 1.2.4-4.el7cp @rhos-10.0-ceph-2.0-mon-signed
libcephfs1.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
librados2.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
librbd1.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
librgw2.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
lttng-ust.x86_64 2.4.1-4.el7cp @rhos-10.0-ceph-2.0-mon-signed
puppet-ceph.noarch 2.3.0-5.el7ost @rhos-10.0-signed
python-cephfs.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
python-flask.noarch 1:0.10.1-5.el7 @rhos-10.0-ceph-2.0-mon-signed
python-jinja2.noarch 2.7.2-2.el7cp @rhos-10.0-ceph-2.0-mon-signed
python-netifaces.x86_64 0.10.4-3.el7ost @rhos-10.0-ceph-2.0-tools-signed
python-rados.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
python-rbd.x86_64 1:10.2.7-28.el7cp @rhos-10.0-ceph-2.0-mon-signed
@rhos-10.0-ceph-2.0-tools-signed
@rhos-10.0-ceph-2.0-tools-signed
userspace-rcu.x86_64 0.7.16-1.el7cp @rhos-10.0-ceph-2.0-mon-signed
Lets check the Ceph cluster status(as user admin) before we begin doing anything. I'll ignore the HEALTH_WARN for the time being. These users are Ceph users and not linux system users. The Ceph users are created using the "ceph auth get-or-create <ceph-user-name>" command
[root@overcloud-controller-0 ~]# ceph -s
cluster 56a3d310-7ba4-11e7-a540-525400ccb563
health HEALTH_WARN
224 pgs degraded
224 pgs stuck degraded
224 pgs stuck unclean
224 pgs stuck undersized
224 pgs undersized
recovery 56/84 objects degraded (66.667%)
monmap e1: 3 mons at {overcloud-controller-0=172.17.3.21:6789/0,overcloud-controller-1=172.17.3.20:6789/0,overcloud-controller-2=172.17.3.15:6789/0}
election epoch 10, quorum 0,1,2 overcloud-controller-2,overcloud-controller-1,overcloud-controller-0
osdmap e20: 2 osds: 2 up, 2 in
flags sortbitwise,require_jewel_osds
pgmap v7155: 224 pgs, 6 pools, 346 kB data, 28 objects
86552 kB used, 30613 MB / 30697 MB avail
56/84 objects degraded (66.667%)
224 active+undersized+degraded
[root@overcloud-controller-0 ~]# ceph osd pool ls
rbd
metrics
images
backups
volumes
vms
[root@overcloud-controller-0 ~]# ceph auth list
installed auth entries:
osd.0
key: AQAAWI5Z0ygRCBAA4p3emGMKLyvpUxuViBK28w==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQACWI5ZIFwIAxAA+59beKFKKmvdg6K6XiBGKg==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQCVu4hZAAAAABAAFCkRGc+Rx6MS8iJx1nrsoQ==
caps: [mds] allow *
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-osd
key: AQCVu4hZAAAAABAAFCkRGc+Rx6MS8iJx1nrsoQ==
caps: [mon] allow profile bootstrap-osd
client.openstack
key: AQCVu4hZAAAAABAAZIqVy2txgF4DfJUyU/6N6A==
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics
The pool created for Glance is "images". Lets look into some details of the "images" pool.
[root@overcloud-controller-0 ~]# ceph osd pool stats images
pool images id 2
nothing is going on
Next, in the /etc/ceph directory we see the keyring for the user "client.openstack" and the ceph.conf files.
[root@overcloud-controller-0 ~]# ls -l /etc/ceph/
total 16
-rw-------. 1 root root 129 Aug 12 01:15 ceph.client.admin.keyring
-rw-r--r--. 1 root root 262 Aug 12 01:15 ceph.client.openstack.keyring
-rw-r--r--. 1 root root 561 Aug 12 01:15 ceph.conf
-rw-r--r--. 1 root root 92 Jul 5 20:22 rbdmap
Lets look into the keyring file for the "openstack" user.
[root@overcloud-controller-0 ~]# cat /etc/ceph/ceph.client.openstack.keyring
[client.openstack]
key = AQCVu4hZAAAAABAAZIqVy2txgF4DfJUyU/6N6A==
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics"
We will no see if we can check the Ceph cluster status as user "openstack".
[root@overcloud-controller-0 ~]# ceph --id openstack -s
cluster 56a3d310-7ba4-11e7-a540-525400ccb563
health HEALTH_WARN
224 pgs degraded
224 pgs stuck degraded
224 pgs stuck unclean
224 pgs stuck undersized
224 pgs undersized
recovery 56/84 objects degraded (66.667%)
monmap e1: 3 mons at {overcloud-controller-0=172.17.3.21:6789/0,overcloud-controller-1=172.17.3.20:6789/0,overcloud-controller-2=172.17.3.15:6789/0}
election epoch 10, quorum 0,1,2 overcloud-controller-2,overcloud-controller-1,overcloud-controller-0
osdmap e20: 2 osds: 2 up, 2 in
flags sortbitwise,require_jewel_osds
pgmap v7159: 224 pgs, 6 pools, 346 kB data, 28 objects
86552 kB used, 30613 MB / 30697 MB avail
56/84 objects degraded (66.667%)
224 active+undersized+degraded
client io 68 B/s rd, 0 op/s rd, 0 op/s wr
Now, just for kicks lets try getting the cluster status as user "foo".
[root@overcloud-controller-0 ~]# ceph --id foo -s
2017-08-16 00:26:11.418444 7f5b36807700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.foo.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2017-08-16 00:26:11.418458 7f5b36807700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2017-08-16 00:26:11.418459 7f5b36807700 0 librados: client.foo initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound
Next, we'll see how Glance is configured to use "rbd" for backend. We can see below that rbd_storage_pool is "images", and "rbd_store_user" is openstack.
[root@overcloud-controller-0 ~]# cat /etc/glance/glance-api.conf | grep rbd
# * rbd
stores = glance.store.http.Store,glance.store.rbd.Store
# * rbd
# Allowed values: file, filesystem, http, https, swift, swift+http, swift+https, swift+config, rbd, sheepdog, cinder, vsphere
default_store = rbd
#rbd_store_chunk_size = 8
#rbd_store_pool = images
rbd_store_pool = images
# section in rbd_store_ceph_conf.
# * rbd_store_ceph_conf
#rbd_store_user = <None>
rbd_store_user = openstack
# * rbd_store_user
#rbd_store_ceph_conf = /etc/ceph/ceph.conf
# * rbd
Next, we can see the status of "openstack-glance-api"
[root@overcloud-controller-0 ~]# systemctl status openstack-glance-api
● openstack-glance-api.service - OpenStack Image Service (code-named Glance) API server
Loaded: loaded (/usr/lib/systemd/system/openstack-glance-api.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2017-08-12 01:48:39 UTC; 3 days ago
Main PID: 190781 (glance-api)
CGroup: /system.slice/openstack-glance-api.service
├─190781 /usr/bin/python2 /usr/bin/glance-api
├─190894 /usr/bin/python2 /usr/bin/glance-api
├─190895 /usr/bin/python2 /usr/bin/glance-api
├─190896 /usr/bin/python2 /usr/bin/glance-api
└─190897 /usr/bin/python2 /usr/bin/glance-api
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
Lets download a cirros image, and then upload it onto Glance.
[root@overcloud-controller-0 ~]# curl -o /tmp/cirros.qcow2 \
> http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12.6M 100 12.6M 0 0 297k 0 0:00:43 0:00:43 --:--:-- 590k
[root@overcloud-controller-0 ~]# openstack image create --disk-format qcow2 --container-format bare --public --file /tmp/cirros.qcow2 cirros
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | ee1eca47dc88f4879d8a229cc70a07c6 |
| container_format | bare |
| created_at | 2017-08-16T01:38:46Z |
| disk_format | qcow2 |
| file | /v2/images/9b8972f9-6116-4bbe-97de-b416a74cdad2/file |
| id | 9b8972f9-6116-4bbe-97de-b416a74cdad2 |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros |
| owner | f0ac0df6e1be446394f28ad66fb40f3c |
| properties | direct_url='rbd://56a3d310-7ba4-11e7-a540-525400ccb563/images/9b8972f9-6116-4bbe-97de-b416a74cdad2/snap', locations='[{u'url': u'rbd://56a3d310-7ba4-11e7-a540-525400ccb563/images/9b8972f9-6116-4bbe- |
| | 97de-b416a74cdad2/snap', u'metadata': {}}]' |
| protected | False |
| schema | /v2/schemas/image |
| size | 13287936 |
| status | active |
| tags | |
| updated_at | 2017-08-16T01:38:53Z |
| virtual_size | None |
| visibility | public |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Using the rbd command we will see that the image name is the "id" of the Glance image.
[root@overcloud-controller-0 ~]# rbd --id openstack -p images ls
9b8972f9-6116-4bbe-97de-b416a74cdad2
The "ceph df" command gives a good detail of all the ceph pools and the usage of each of them.
For more details on the osd. The ceph-osd did not start for some reason on overcloud-cephstorage-1, I need to root cause that.
[root@overcloud-cephstorage-0 ceph]# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
30697M 30582M 115M 0.38
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 10194M 0
metrics 1 609k 0 10194M 47
images 2 12976k 0.04 10194M 7
backups 3 0 0 10194M 0
volumes 4 0 0 10194M 0
vms 5 0 0 10194M 0
For more details on the osd. The ceph-osd did not start for some reason on overcloud-cephstorage-1, I need to root cause that.
[root@overcloud-cephstorage-0 ceph]# ceph osd df tree
ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME
-1 0.02917 - 30697M 115M 30582M 0.38 1.00 0 root default
-2 0.01459 - 15348M 59464k 15290M 0.38 1.00 0 host overcloud-cephstorage-2
0 0.01459 1.00000 15348M 59464k 15290M 0.38 1.00 224 osd.0
-3 0.01459 - 15348M 59228k 15291M 0.38 1.00 0 host overcloud-cephstorage-0
1 0.01459 1.00000 15348M 59228k 15291M 0.38 1.00 224 osd.1
TOTAL 30697M 115M 30582M 0.38
MIN/MAX VAR: 1.00/1.00 STDDEV: 0
I have 1 x HDD on each of the Ceph Storage nodes dedicated to Ceph, so for that reason we have both data and journal on the same disk. The output of "fdisk -l" clearly shows that the first partition of vdb is used for data, while the second partition is used for journal.
[root@overcloud-cephstorage-0 ~]# fdisk -l
Disk /dev/vda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000451ac
Device Boot Start End Blocks Id System
/dev/vda1 2048 4095 1024 83 Linux
/dev/vda2 * 4096 41943006 20969455+ 83 Linux
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/vdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
Disk identifier: EF24FDED-84D2-4723-A855-092A9438F6F7
# Start End Size Type Name
1 10487808 41943006 15G unknown ceph data
2 2048 10487807 5G unknown ceph journal
No comments:
Post a Comment