Thursday, November 30, 2017

Hardware requirements for Red Hat OpenStack and Red Ceph Storage

I recently worked with a couple of our OEM Partners who wanted to create a Reference Architecture with Red Hat OpenStack, and Red Hat Ceph Storage. One of the first questions that come up during the conversation is the hardware requirement for a Reference Architecture.

The below presentation covers not only the minimum requirements for an OpenStack deployment with Ceph Storage, but also the hardware(storage node) requirements for various Ceph workloads like IOPS-Optimized, Throughput-Optimized, and Cost/Capacity Optimized workloads.


  

Tuesday, August 15, 2017

Ceph for Glance in Red Hat OpenStack

In the previous post we saw the deployment of OpenStack Platform 10 with Ceph.

Lets log into the Controller Node, and look at all the services running.

[root@overcloud-controller-0 ~]# openstack service list
+----------------------------------+------------+----------------+
| ID                               | Name       | Type           |
+----------------------------------+------------+----------------+
| 316642206a194dc58cb4528adbc90f5d | heat-cfn   | cloudformation |
| 593f9fde36634401977d2806a9e6f6f5 | gnocchi    | metric         |
| 6fd62024660942ec8d6451429f03d336 | ceilometer | metering       |
| 762a53b0efd54947b3d6d2d6405440b0 | cinderv2   | volumev2       |
| 7783685d418845b9acc1bad50fbbfe7d | nova       | compute        |
| 93987d0d0fea4f02a73e7825c9b4adfe | glance     | image          |
| 9a9990c7780945aca10fbeba19cc4729 | aodh       | alarming       |
| a13c6596df864b17a33dc674a278c31c | cinderv3   | volumev3       |
| bdb018992e1146b499d5b56d8fdfce29 | heat       | orchestration  |
| d0e8d3c58784485888d5b36d71af39a4 | keystone   | identity       |
| d7a05ca55d2349d2b1f4dbc5fb79fb2d | cinder     | volume         |
| e238bbf05e8547e7b4b57ee4dba52a63 | neutron    | network        |
| fb32e87e977742899426a848077b09db | swift      | object-store   |
+----------------------------------+------------+----------------+

Lets now look at all the Ceph related packages installed on the Controller Node.

[root@overcloud-controller-0 ~]# yum list installed | grep -i ceph
ceph-base.x86_64                    1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
ceph-common.x86_64                  1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
ceph-mon.x86_64                     1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
ceph-osd.x86_64                     1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-osd-signed
ceph-radosgw.x86_64                 1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-tools-signed
ceph-selinux.x86_64                 1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
fcgi.x86_64                         2.4.0-27.el7cp     @rhos-10.0-ceph-2.0-mon-signed
gperftools-libs.x86_64              2.4-8.el7          @rhos-10.0-ceph-2.0-mon-signed
leveldb.x86_64                      1.12.0-7.el7cp     @rhos-10.0-ceph-2.0-mon-signed
libbabeltrace.x86_64                1.2.4-4.el7cp      @rhos-10.0-ceph-2.0-mon-signed
libcephfs1.x86_64                   1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
librados2.x86_64                    1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
librbd1.x86_64                      1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
librgw2.x86_64                      1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
lttng-ust.x86_64                    2.4.1-4.el7cp      @rhos-10.0-ceph-2.0-mon-signed
puppet-ceph.noarch                  2.3.0-5.el7ost     @rhos-10.0-signed        
python-cephfs.x86_64                1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
python-flask.noarch                 1:0.10.1-5.el7     @rhos-10.0-ceph-2.0-mon-signed
python-jinja2.noarch                2.7.2-2.el7cp      @rhos-10.0-ceph-2.0-mon-signed
python-netifaces.x86_64             0.10.4-3.el7ost    @rhos-10.0-ceph-2.0-tools-signed
python-rados.x86_64                 1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
python-rbd.x86_64                   1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed
                                                       @rhos-10.0-ceph-2.0-tools-signed
                                                       @rhos-10.0-ceph-2.0-tools-signed
userspace-rcu.x86_64                0.7.16-1.el7cp     @rhos-10.0-ceph-2.0-mon-signed

Lets check the Ceph cluster status(as user admin) before we begin doing anything. I'll ignore the HEALTH_WARN for the time being. These users are Ceph users and not linux system users. The Ceph users are created using the "ceph auth get-or-create <ceph-user-name>" command

[root@overcloud-controller-0 ~]# ceph -s
    cluster 56a3d310-7ba4-11e7-a540-525400ccb563
     health HEALTH_WARN
            224 pgs degraded
            224 pgs stuck degraded
            224 pgs stuck unclean
            224 pgs stuck undersized
            224 pgs undersized
            recovery 56/84 objects degraded (66.667%)
     monmap e1: 3 mons at {overcloud-controller-0=172.17.3.21:6789/0,overcloud-controller-1=172.17.3.20:6789/0,overcloud-controller-2=172.17.3.15:6789/0}
            election epoch 10, quorum 0,1,2 overcloud-controller-2,overcloud-controller-1,overcloud-controller-0
     osdmap e20: 2 osds: 2 up, 2 in
            flags sortbitwise,require_jewel_osds
      pgmap v7155: 224 pgs, 6 pools, 346 kB data, 28 objects
            86552 kB used, 30613 MB / 30697 MB avail
            56/84 objects degraded (66.667%)
                 224 active+undersized+degraded

Lets verify if the required Ceph configurations are created to integrate with Glance. We will see that a Ceph user "client.openstack" with read permission for monitor, and read, write, execute permission for the created pool.

[root@overcloud-controller-0 ~]# ceph osd pool ls
rbd
metrics
images
backups
volumes
vms

[root@overcloud-controller-0 ~]# ceph auth list
installed auth entries:

osd.0
key: AQAAWI5Z0ygRCBAA4p3emGMKLyvpUxuViBK28w==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQACWI5ZIFwIAxAA+59beKFKKmvdg6K6XiBGKg==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQCVu4hZAAAAABAAFCkRGc+Rx6MS8iJx1nrsoQ==
caps: [mds] allow *
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-osd
key: AQCVu4hZAAAAABAAFCkRGc+Rx6MS8iJx1nrsoQ==
caps: [mon] allow profile bootstrap-osd
client.openstack
key: AQCVu4hZAAAAABAAZIqVy2txgF4DfJUyU/6N6A==
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics

The pool created for Glance is "images". Lets look into some details of the "images" pool.

[root@overcloud-controller-0 ~]# ceph osd pool stats images
pool images id 2
  nothing is going on



Next, in the /etc/ceph directory we see the keyring for the user "client.openstack" and the ceph.conf files.

[root@overcloud-controller-0 ~]# ls -l /etc/ceph/
total 16
-rw-------. 1 root root 129 Aug 12 01:15 ceph.client.admin.keyring
-rw-r--r--. 1 root root 262 Aug 12 01:15 ceph.client.openstack.keyring
-rw-r--r--. 1 root root 561 Aug 12 01:15 ceph.conf
-rw-r--r--. 1 root root  92 Jul  5 20:22 rbdmap

Lets look into the keyring file for the "openstack" user.

[root@overcloud-controller-0 ~]# cat /etc/ceph/ceph.client.openstack.keyring 
[client.openstack]
key = AQCVu4hZAAAAABAAZIqVy2txgF4DfJUyU/6N6A==
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics"

We will no see if we can check the Ceph cluster status as user "openstack".

[root@overcloud-controller-0 ~]# ceph --id openstack -s
    cluster 56a3d310-7ba4-11e7-a540-525400ccb563
     health HEALTH_WARN
            224 pgs degraded
            224 pgs stuck degraded
            224 pgs stuck unclean
            224 pgs stuck undersized
            224 pgs undersized
            recovery 56/84 objects degraded (66.667%)
     monmap e1: 3 mons at {overcloud-controller-0=172.17.3.21:6789/0,overcloud-controller-1=172.17.3.20:6789/0,overcloud-controller-2=172.17.3.15:6789/0}
            election epoch 10, quorum 0,1,2 overcloud-controller-2,overcloud-controller-1,overcloud-controller-0
     osdmap e20: 2 osds: 2 up, 2 in
            flags sortbitwise,require_jewel_osds
      pgmap v7159: 224 pgs, 6 pools, 346 kB data, 28 objects
            86552 kB used, 30613 MB / 30697 MB avail
            56/84 objects degraded (66.667%)
                 224 active+undersized+degraded
  client io 68 B/s rd, 0 op/s rd, 0 op/s wr

Now, just for kicks lets try getting the cluster status as user "foo".

[root@overcloud-controller-0 ~]# ceph --id foo -s
2017-08-16 00:26:11.418444 7f5b36807700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.foo.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2017-08-16 00:26:11.418458 7f5b36807700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2017-08-16 00:26:11.418459 7f5b36807700  0 librados: client.foo initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound

Next, we'll see how Glance is configured to use "rbd" for backend. We can see below that rbd_storage_pool is "images", and "rbd_store_user" is openstack.

[root@overcloud-controller-0 ~]# cat /etc/glance/glance-api.conf | grep rbd
#         * rbd
stores = glance.store.http.Store,glance.store.rbd.Store
#     * rbd
# Allowed values: file, filesystem, http, https, swift, swift+http, swift+https, swift+config, rbd, sheepdog, cinder, vsphere
default_store = rbd
#rbd_store_chunk_size = 8
#rbd_store_pool = images
rbd_store_pool = images
# section in rbd_store_ceph_conf.
#     * rbd_store_ceph_conf
#rbd_store_user = <None>
rbd_store_user = openstack
#     * rbd_store_user
#rbd_store_ceph_conf = /etc/ceph/ceph.conf
#         * rbd

Next, we can see the status of "openstack-glance-api"

[root@overcloud-controller-0 ~]# systemctl status openstack-glance-api
openstack-glance-api.service - OpenStack Image Service (code-named Glance) API server
   Loaded: loaded (/usr/lib/systemd/system/openstack-glance-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2017-08-12 01:48:39 UTC; 3 days ago
 Main PID: 190781 (glance-api)
   CGroup: /system.slice/openstack-glance-api.service
           ├─190781 /usr/bin/python2 /usr/bin/glance-api
           ├─190894 /usr/bin/python2 /usr/bin/glance-api
           ├─190895 /usr/bin/python2 /usr/bin/glance-api
           ├─190896 /usr/bin/python2 /usr/bin/glance-api
           └─190897 /usr/bin/python2 /usr/bin/glance-api

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Lets download a cirros image, and then upload it onto Glance.

[root@overcloud-controller-0 ~]# curl -o /tmp/cirros.qcow2 \
>  http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.6M  100 12.6M    0     0   297k      0  0:00:43  0:00:43 --:--:--  590k


[root@overcloud-controller-0 ~]# openstack image create  --disk-format qcow2 --container-format bare --public  --file /tmp/cirros.qcow2 cirros
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                                                                                      |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum         | ee1eca47dc88f4879d8a229cc70a07c6                                                                                                                                                                           |
| container_format | bare                                                                                                                                                                                                       |
| created_at       | 2017-08-16T01:38:46Z                                                                                                                                                                                       |
| disk_format      | qcow2                                                                                                                                                                                                      |
| file             | /v2/images/9b8972f9-6116-4bbe-97de-b416a74cdad2/file                                                                                                                                                       |
| id               | 9b8972f9-6116-4bbe-97de-b416a74cdad2                                                                                                                                                                       |
| min_disk         | 0                                                                                                                                                                                                          |
| min_ram          | 0                                                                                                                                                                                                          |
| name             | cirros                                                                                                                                                                                                     |
| owner            | f0ac0df6e1be446394f28ad66fb40f3c                                                                                                                                                                           |
| properties       | direct_url='rbd://56a3d310-7ba4-11e7-a540-525400ccb563/images/9b8972f9-6116-4bbe-97de-b416a74cdad2/snap', locations='[{u'url': u'rbd://56a3d310-7ba4-11e7-a540-525400ccb563/images/9b8972f9-6116-4bbe-     |
|                  | 97de-b416a74cdad2/snap', u'metadata': {}}]'                                                                                                                                                                |
| protected        | False                                                                                                                                                                                                      |
| schema           | /v2/schemas/image                                                                                                                                                                                          |
| size             | 13287936                                                                                                                                                                                                   |
| status           | active                                                                                                                                                                                                     |
| tags             |                                                                                                                                                                                                            |
| updated_at       | 2017-08-16T01:38:53Z                                                                                                                                                                                       |
| virtual_size     | None                                                                                                                                                                                                       |
| visibility       | public                                                                                                                                                                                                     |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Using the rbd command we will see that the image name is the "id" of the Glance image. 

[root@overcloud-controller-0 ~]# rbd --id openstack -p images ls
9b8972f9-6116-4bbe-97de-b416a74cdad2

The "ceph df" command gives a good detail of all the ceph pools and the usage of each of them.

[root@overcloud-cephstorage-0 ceph]# ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    30697M     30582M         115M          0.38 
POOLS:
    NAME        ID     USED       %USED     MAX AVAIL     OBJECTS 
    rbd         0           0         0        10194M           0 
    metrics     1        609k         0        10194M          47 
    images      2      12976k      0.04        10194M           7 
    backups     3           0         0        10194M           0 
    volumes     4           0         0        10194M           0 
    vms         5           0         0        10194M           0 

For more details on the osd. The ceph-osd did not start for some reason on overcloud-cephstorage-1, I need to root cause that.

[root@overcloud-cephstorage-0 ceph]# ceph osd df tree
ID WEIGHT  REWEIGHT SIZE   USE    AVAIL  %USE VAR  PGS TYPE NAME                        
-1 0.02917        - 30697M   115M 30582M 0.38 1.00   0 root default                     
-2 0.01459        - 15348M 59464k 15290M 0.38 1.00   0     host overcloud-cephstorage-2 
 0 0.01459  1.00000 15348M 59464k 15290M 0.38 1.00 224         osd.0                    
-3 0.01459        - 15348M 59228k 15291M 0.38 1.00   0     host overcloud-cephstorage-0 
 1 0.01459  1.00000 15348M 59228k 15291M 0.38 1.00 224         osd.1                    
              TOTAL 30697M   115M 30582M 0.38                                           

MIN/MAX VAR: 1.00/1.00  STDDEV: 0

I have 1 x HDD on each of the Ceph Storage nodes dedicated to Ceph, so for that reason we have both data and journal on the same disk. The output of "fdisk -l" clearly shows that the first partition of vdb is used for data, while the second partition is used for journal.

[root@overcloud-cephstorage-0 ~]# fdisk -l

Disk /dev/vda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000451ac

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1            2048        4095        1024   83  Linux
/dev/vda2   *        4096    41943006    20969455+  83  Linux
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/vdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
Disk identifier: EF24FDED-84D2-4723-A855-092A9438F6F7


#         Start          End    Size  Type            Name
 1     10487808     41943006     15G  unknown         ceph data
 2         2048     10487807      5G  unknown         ceph journal



Red Hat OpenStack 10 deployment with Ceph Storage

I had deployed Red Hat OpenStack Platform 10 (Neuton) on my Dell box using Virtual Machines.

The deployment had 3 x Controller Nodes, 1 x Compute Node, and 3 x Ceph Nodes


[root@openstack ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 249   undercloud                     running
 278   overcloud-controller3          running
 279   overcloud-ceph2                running
 280   overcloud-compute1             running
 281   overcloud-ceph3                running
 282   overcloud-ceph1                running
 285   overcloud-controller2          running
 286   overcloud-controller1          running
 -     overcloud-compute2             shut off

I will not go through the entire deployment process, but will show some of the yaml files needed for the deployment.

Here is the command/script that was used to deploy the Overcloud.

[stack@undercloud ~]$ cat run_deploy.sh.ceph 
#!/bin/bash

set -o verbose
source stackrc
pushd /home/stack

openstack overcloud deploy --templates ~/my_templates -e /home/stack/my_templates/advanced-networking.yaml -e /home/stack/my_templates/storage-environment.yaml --ntp-server 0.north-america.pool.ntp.org --control-flavor control --control-scale 3 --compute-flavor compute --compute-scale 1 --ceph-storage-flavor ceph-storage --ceph-storage-scale 3 --neutron-tunnel-types vxlan --neutron-network-type vxlan

echo DONE

Next, we will look at the storage-environment.yaml file used in this deployment.

[stack@undercloud ~]$ cat my_templates/storage-environment.yaml
## A Heat environment file which can be used to set up storage
## backends. Defaults to Ceph used as a backend for Cinder, Glance and
## Nova ephemeral storage.
resource_registry:
  OS::TripleO::Services::CephMon: /usr/share/openstack-tripleo-heat-templates/puppet/services/ceph-mon.yaml
  OS::TripleO::Services::CephOSD: /usr/share/openstack-tripleo-heat-templates/puppet/services/ceph-osd.yaml
  OS::TripleO::Services::CephClient: /usr/share/openstack-tripleo-heat-templates/puppet/services/ceph-client.yaml

  OS::TripleO::NodeUserData: /home/stack/my_templates/firstboot/wipe-disks.yaml

parameter_defaults:

  #### BACKEND SELECTION ####

  ## Whether to enable iscsi backend for Cinder.
  CinderEnableIscsiBackend: false
  ## Whether to enable rbd (Ceph) backend for Cinder.
  CinderEnableRbdBackend: true
  ## Cinder Backup backend can be either 'ceph' or 'swift'.
  CinderBackupBackend: ceph
  ## Whether to enable NFS backend for Cinder.
  # CinderEnableNfsBackend: false
  ## Whether to enable rbd (Ceph) backend for Nova ephemeral storage.
  NovaEnableRbdBackend: true
  ## Glance backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GlanceBackend: rbd
  ## Gnocchi backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GnocchiBackend: rbd


  #### CINDER NFS SETTINGS ####

  ## NFS mount options
  # CinderNfsMountOptions: ''
  ## NFS mount point, e.g. '192.168.122.1:/export/cinder'
  # CinderNfsServers: ''


  #### GLANCE NFS SETTINGS ####

  ## Make sure to set `GlanceBackend: file` when enabling NFS
  ##
  ## Whether to make Glance 'file' backend a NFS mount
  # GlanceNfsEnabled: false
  ## NFS share for image storage, e.g. '192.168.122.1:/export/glance'
  ## (If using IPv6, use both double- and single-quotes,
  ## e.g. "'[fdd0::1]:/export/glance'")
  # GlanceNfsShare: ''
  ## Mount options for the NFS image storage mount point
  # GlanceNfsOptions: 'intr,context=system_u:object_r:glance_var_lib_t:s0'


  #### CEPH SETTINGS ####

  ## When deploying Ceph Nodes through the oscplugin CLI, the following
  ## parameters are set automatically by the CLI. When deploying via
  ## heat stack-create or ceph on the controller nodes only,
  ## they need to be provided manually.

  ## Number of Ceph storage nodes to deploy
  # CephStorageCount: 0
  ## Ceph FSID, e.g. '4b5c8c0a-ff60-454b-a1b4-9747aa737d19'
  # CephClusterFSID: ''
  ## Ceph monitor key, e.g. 'AQC+Ox1VmEr3BxAALZejqeHj50Nj6wJDvs96OQ=='
  # CephMonKey: ''
  ## Ceph admin key, e.g. 'AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ=='
  # CephAdminKey: ''
  ## Ceph client key, e.g 'AQC+vYNXgDAgAhAAc8UoYt+OTz5uhV7ItLdwUw=='
  # CephClientKey: ''

  ExtraConfig:
    ceph::profile::params::osds:
      '/dev/vdb': {}


As you see above rbd(Ceph) is used as the backend for Cinder. Ceph is also used for Cinder backup, for Nova ephemeral storage, for Glance backend to store images, and for Gnocchi backend.

There are two other files that I'd like to share, the first one is the advanced_network.yam and the other being the nic-config file for the Ceph Node.

[stack@undercloud ~]$ cat my_templates/advanced-networking.yaml
# Enable the creation of Neutron networks for isolated Overcloud
# traffic and configure each role to assign ports (related
# to that role) on these networks.
resource_registry:
  OS::TripleO::Network::External: /home/stack/my_templates/network/external.yaml
  OS::TripleO::Network::InternalApi: /home/stack/my_templates/network/internal_api.yaml
  OS::TripleO::Network::StorageMgmt: /home/stack/my_templates/network/storage_mgmt.yaml
  OS::TripleO::Network::Storage: /home/stack/my_templates/network/storage.yaml
  OS::TripleO::Network::Tenant: /home/stack/my_templates/network/tenant.yaml
  # Management network is optional and disabled by default.
  # To enable it, include environments/network-management.yaml
  #OS::TripleO::Network::Management: /home/stack/my_templates/network/management.yaml

  # Port assignments for the VIPs
  OS::TripleO::Network::Ports::ExternalVipPort: /home/stack/my_templates/network/ports/external.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: /home/stack/my_templates/network/ports/internal_api.yaml
  OS::TripleO::Network::Ports::StorageVipPort: /home/stack/my_templates/network/ports/storage.yaml
  OS::TripleO::Network::Ports::StorageMgmtVipPort: /home/stack/my_templates/network/ports/storage_mgmt.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /home/stack/my_templates/network/ports/vip.yaml

  # Port assignments for service virtual IPs for the controller role
  OS::TripleO::Controller::Ports::RedisVipPort: /home/stack/my_templates/network/ports/vip.yaml
  # Port assignments for the controller role
  OS::TripleO::Controller::Ports::ExternalPort: /home/stack/my_templates/network/ports/external.yaml
  OS::TripleO::Controller::Ports::InternalApiPort: /home/stack/my_templates/network/ports/internal_api.yaml
  OS::TripleO::Controller::Ports::StoragePort: /home/stack/my_templates/network/ports/storage.yaml
  OS::TripleO::Controller::Ports::StorageMgmtPort: /home/stack/my_templates/network/ports/storage_mgmt.yaml
  OS::TripleO::Controller::Ports::TenantPort: /home/stack/my_templates/network/ports/tenant.yaml
  #OS::TripleO::Controller::Ports::ManagementPort: /home/stack/my_templates/network/ports/management.yaml

  # Port assignments for the compute role
  OS::TripleO::Compute::Ports::ExternalPort: /home/stack/my_templates/network/ports/noop.yaml
  OS::TripleO::Compute::Ports::InternalApiPort: /home/stack/my_templates/network/ports/internal_api.yaml
  OS::TripleO::Compute::Ports::StoragePort: /home/stack/my_templates/network/ports/storage.yaml
  OS::TripleO::Compute::Ports::StorageMgmtPort: /home/stack/my_templates/network/ports/noop.yaml
  OS::TripleO::Compute::Ports::TenantPort: /home/stack/my_templates/network/ports/tenant.yaml
  #OS::TripleO::Compute::Ports::ManagementPort: /home/stack/my_templates/network/ports/management.yaml

  # Port assignments for the ceph storage role
  OS::TripleO::CephStorage::Ports::ExternalPort: /home/stack/my_templates/network/ports/noop.yaml
  OS::TripleO::CephStorage::Ports::InternalApiPort: /home/stack/my_templates/network/ports/noop.yaml
  OS::TripleO::CephStorage::Ports::StoragePort: /home/stack/my_templates/network/ports/storage.yaml
  OS::TripleO::CephStorage::Ports::StorageMgmtPort: /home/stack/my_templates/network/ports/storage_mgmt.yaml
  OS::TripleO::CephStorage::Ports::TenantPort: /home/stack/my_templates/network/ports/noop.yaml
  #OS::TripleO::CephStorage::Ports::ManagementPort: /home/stack/my_templates/network/ports/management.yaml


# NIC Configs for our roles
  OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/my_templates/nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/my_templates/nic-configs/controller.yaml
  OS::TripleO::CephStorage::Net::SoftwareConfig: /home/stack/my_templates/nic-configs/ceph-storage.yaml

# Enable the creation of Neutron networks for isolated Overcloud
# traffic and configure each role to assign ports (related
# to that role) on these networks.

parameter_defaults:
  # Internal API used for private OpenStack Traffic
  InternalApiNetCidr: 172.17.1.0/24
  InternalApiAllocationPools: [{'start': '172.17.1.10', 'end': '172.17.1.200'}]
  InternalApiNetworkVlanID: 101

  # Tenant Network Traffic - will be used for VXLAN over VLAN
  TenantNetCidr: 172.17.2.0/24
  TenantAllocationPools: [{'start': '172.17.2.10', 'end': '172.17.2.200'}]
  TenantNetworkVlanID: 201

  StorageNetCidr: 172.17.3.0/24
  StorageAllocationPools: [{'start': '172.17.3.10', 'end': '172.17.3.200'}]
  StorageNetworkVlanID: 301

  StorageMgmtNetCidr: 172.17.4.0/24
  StorageMgmtAllocationPools: [{'start': '172.17.4.10', 'end': '172.17.4.200'}]
  StorageMgmtNetworkVlanID: 401

  # External Networking Access - Public API Access
  ExternalNetCidr: 192.168.122.0/24

  # Leave room for floating IPs in the External allocation pool (if required)
  ExternalAllocationPools: [{'start': '192.168.122.100', 'end': '192.168.122.129'}]

  # Set to the router gateway on the external network
  ExternalInterfaceDefaultRoute: 192.168.122.1

  # Add in configuration for the Control Plane
  ControlPlaneSubnetCidr: "24"
  ControlPlaneDefaultRoute: 172.16.0.1
  EC2MetadataIp: 172.16.0.1
  DnsServers: ['192.168.122.1', '8.8.8.8']


Next, we will look at the NIC config for the Ceph Storage Node

[stack@undercloud ~]$ cat /home/stack/my_templates/nic-configs/ceph-storage.yaml
heat_template_version: 2015-04-30

description: >
  Software Config to drive os-net-config to configure multiple interfaces
  for the ceph storage role.

parameters:
  ControlPlaneIp:
    default: ''
    description: IP address/subnet on the ctlplane network
    type: string
  ExternalIpSubnet:
    default: ''
    description: IP address/subnet on the external network
    type: string
  InternalApiIpSubnet:
    default: ''
    description: IP address/subnet on the internal API network
    type: string
  StorageIpSubnet:
    default: ''
    description: IP address/subnet on the storage network
    type: string
  StorageMgmtIpSubnet:
    default: ''
    description: IP address/subnet on the storage mgmt network
    type: string
  TenantIpSubnet:
    default: ''
    description: IP address/subnet on the tenant network
    type: string
  ManagementIpSubnet: # Only populated when including environments/network-management.yaml
    default: ''
    description: IP address/subnet on the management network
    type: string
  ExternalNetworkVlanID:
    default: 10
    description: Vlan ID for the external network traffic.
    type: number
  InternalApiNetworkVlanID:
    default: 20
    description: Vlan ID for the internal_api network traffic.
    type: number
  StorageNetworkVlanID:
    default: 30
    description: Vlan ID for the storage network traffic.
    type: number
  StorageMgmtNetworkVlanID:
    default: 40
    description: Vlan ID for the storage mgmt network traffic.
    type: number
  TenantNetworkVlanID:
    default: 50
    description: Vlan ID for the tenant network traffic.
    type: number
  ExternalInterfaceDefaultRoute:
    default: '10.0.0.1'
    description: default route for the external network
    type: string
  ControlPlaneSubnetCidr: # Override this via parameter_defaults
    default: '24'
    description: The subnet CIDR of the control plane network.
    type: string
  ControlPlaneDefaultRoute: # Override this via parameter_defaults
    description: The subnet CIDR of the control plane network.
    type: string
  DnsServers: # Override this via parameter_defaults
    default: []
    description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf.
    type: json
  EC2MetadataIp: # Override this via parameter_defaults
    description: The IP address of the EC2 metadata server.
    type: string

resources:
  OsNetConfigImpl:
    type: OS::Heat::StructuredConfig
    properties:
      group: os-apply-config
      config:
        os_net_config:
          network_config:
            -
              type: interface
              name: nic1
              use_dhcp: false
              dns_servers: {get_param: DnsServers}
              addresses:
                -
                  ip_netmask:
                    list_join:
                      - '/'
                      - - {get_param: ControlPlaneIp}
                        - {get_param: ControlPlaneSubnetCidr}
              routes:
                -
                  ip_netmask: 0.0.0.0/0
                  next_hop: {get_param: ControlPlaneDefaultRoute}
                  # Optionally have this interface as default route
                  default: true
                -
                  ip_netmask: 169.254.169.254/32
                  next_hop: {get_param: EC2MetadataIp}
            -
              type: ovs_bridge
              name: br-isolated
              use_dhcp: false
              members:
                -
                  type: interface
                  name: nic2
                  # force the MAC address of the bridge to this interface
                  primary: true
                -
                  type: vlan
                  vlan_id: {get_param: StorageNetworkVlanID}
                  addresses:
                  -
                    ip_netmask: {get_param: StorageIpSubnet}
                -
                  type: vlan
                  vlan_id: {get_param: StorageMgmtNetworkVlanID}
                  addresses:
                  -
                    ip_netmask: {get_param: StorageMgmtIpSubnet}

outputs:
  OS::stack_id:
    description: The OsNetConfigImpl resource.
    value: {get_resource: OsNetConfigImpl}

Nic 1 of the Ceph node was used for Provisioning, which Nic 2 is vlan-ed for Storage and Storage Magement.

 After the deployment completes successfully, here is what you will see.

[stack@undercloud ~]$ neutron net-list
+--------------------------------------+--------------+-------------------------------------------------------+
| id                                   | name         | subnets                                               |
+--------------------------------------+--------------+-------------------------------------------------------+
| 542b5f6d-d0a6-4fd2-a905-996208a77525 | storage      | 4449344f-089f-426f-bd5f-37591e9e71ea 172.17.3.0/24    |
| 9e2ab19a-3b29-47a7-8db8-995883e1511b | internal_api | 1901412a-d54b-4274-8042-7daacfe07bfc 172.17.1.0/24    |
| a9fedf2d-537c-461c-b15c-1f3fe2467172 | ctlplane     | 869c6ad4-d133-4076-875f-347026dfee88 172.16.0.0/24    |
| b7bc5c80-a57b-40b7-ac5c-9beac1da84d7 | storage_mgmt | 2b724ccd-a9eb-4aa8-907a-14bcebd34e27 172.17.4.0/24    |
| c5392029-9438-4521-8202-f9a89648d8a5 | tenant       | 3b4b55a3-97d1-40dc-b95c-51bd2012e2ae 172.17.2.0/24    |
| cef77887-476d-47ba-8d06-4fa65acbe5ff | external     | 0fca9fa0-b3eb-4b25-a987-cc936a28d659 192.168.122.0/24 |
+--------------------------------------+--------------+-------------------------------------------------------+
[stack@undercloud ~]$ ironic node-list
+--------------------------------------+-----------------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name                  | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------------------+--------------------------------------+-------------+--------------------+-------------+
| 945a30f3-dc7f-4ca4-946d-18f38a352e1e | overcloud-ceph1       | bdc05172-4e5f-4aaf-9758-bbe0d2cb2012 | power on    | active             | False       |
| d78b8cac-4379-4825-bb22-4e9e9f259fcd | overcloud-ceph2       | 1208a491-157f-4f43-9267-017aba21b331 | power on    | active             | False       |
| 4b50450a-eaee-4828-ab43-a96f36d52789 | overcloud-ceph3       | 2a658d35-64f7-40d4-9d54-142640279a86 | power on    | active             | False       |
| 682b6813-5df2-4829-927a-a1b95e081119 | overcloud-compute1    | 8b626a49-ad7b-4e1e-9dec-8ca1abb5073e | power on    | active             | False       |
| 2645e0ed-c403-4e16-afdd-b293726fd0eb | overcloud-compute2    | None                                 | power off   | available          | False       |
| af20af15-7d13-41fa-9862-2a191108be4c | overcloud-controller1 | 5ee03387-25bc-4368-9da4-e6a73cecd455 | power on    | active             | False       |
| 4272e8f0-7409-4858-b17a-323ff0fbd43a | overcloud-controller2 | f55665f3-8926-4bc2-8068-74d9001600e6 | power on    | active             | False       |
| a9325cf0-9b0f-405f-8cbe-697e4146ffbd | overcloud-controller3 | 40fc318e-22fe-4418-a063-eae58719da38 | power on    | active             | False       |
+--------------------------------------+-----------------------+--------------------------------------+-------------+--------------------+-------------+
[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+----------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks             |
+--------------------------------------+-------------------------+--------+------------+-------------+----------------------+
| bdc05172-4e5f-4aaf-9758-bbe0d2cb2012 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=172.16.0.32 |
| 1208a491-157f-4f43-9267-017aba21b331 | overcloud-cephstorage-1 | ACTIVE | -          | Running     | ctlplane=172.16.0.26 |
| 2a658d35-64f7-40d4-9d54-142640279a86 | overcloud-cephstorage-2 | ACTIVE | -          | Running     | ctlplane=172.16.0.34 |
| 8b626a49-ad7b-4e1e-9dec-8ca1abb5073e | overcloud-compute-0     | ACTIVE | -          | Running     | ctlplane=172.16.0.27 |
| 40fc318e-22fe-4418-a063-eae58719da38 | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=172.16.0.31 |
| 5ee03387-25bc-4368-9da4-e6a73cecd455 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=172.16.0.23 |
| f55665f3-8926-4bc2-8068-74d9001600e6 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=172.16.0.36 |
+--------------------------------------+-------------------------+--------+------------+-------------+----------------------+

In the next post we will look into the details of Ceph.