In this article we will show you how to install OpenShift in mutliple nodes using a quick install command, atomic-openshift-installer, which is powered by ansible.

Host preparation

I use under virtual machines for OpenShift nodes to deploy

TypeCPUMemHDDhostnameOS
Master12 GB20 GBmaster.example.comRHEL 7
node12 GB20 GBnode1.example.comRHEL 7
node12 GB20 GBnode2.example.comRHEL 7

Host Registration

Note: If you are using other OS but not RHEL, Please go to [# Install necessary packages](# Install necessary packages) to install the packages. If you can not install some of them, try to add some repos or get the rpm file.

Register each host with RHSM(Red Hat Subscription Manager) to access the required packages.

  1. Register with RHSM for each host:

    1
    
    # subscription-manager register --username=<user_name> --password=<password>
    
  2. List the available OpenShift subscriptions:

    1
    
    # subscription-manager list --available --matches '*OpenShift*'
    
  3. Find pool ID for an OpenShift Container Platform subscription and attach it.

    1
    
    # subscription-manager attach --pool=<pool_id>
    
  4. Disable all repositories and enable only the repositories required by OpenShift Container Platform 3.5

    1
    2
    3
    4
    5
    6
    7
    
    # subscription-manager repos --disable="*"
    # yum-config-manager --disable \*
    # subscription-manager repos \
        --enable="rhel-7-server-rpms" \
        --enable="rhel-7-server-extras-rpms" \
        --enable="rhel-7-server-ose-3.5-rpms" \
        --enable="rhel-7-fast-datapath-rpms"
    

Install necessary packages

Install the following packages.

1
2
3
4
# yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion kexec sos psacct
# yum update
# yum -y install atomic-openshift-utils atomic-openshift-excluder atomic-openshift-docker-excluder
# atomic-openshift-excluder unexclude

Install and configure docker

Install docker

1
# yum -y install docker

Add parameter to docker configuration file

Edit /etc/sysconfig/docker file and add --insecure-registry 172.30.0.0/16 to the OPTIONS parameter.

1
OPTIONS='--selinux-enabled --insecure-registry 172.30.0.0/16'

Configure Docker Storage

Here we use an additional block device for docker storage. In /etc/sysconfig/docker-storage-setup , set DEVS to the path of the disk device. Set VG to the volume group name you wish to create.

1
2
3
4
# cat <<EOF > /etc/sysconfig/docker-storage-setup
DEVS=/dev/vdc
VG=docker-vg
EOF

Then run docker-storage-setup and check to make sure the docker-vg was created.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# docker-storage-setup                                                                                                                                                                                                                                [5/1868]
0
Checking that no-one is using this disk right now ...
OK

Disk /dev/vdc: 31207 cylinders, 16 heads, 63 sectors/track
sfdisk:  /dev/vdc: unrecognized partition table type

Old situation:
sfdisk: No partitions found

New situation:
Units: sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/vdc1          2048  31457279   31455232  8e  Linux LVM
/dev/vdc2             0         -          0   0  Empty
/dev/vdc3             0         -          0   0  Empty
/dev/vdc4             0         -          0   0  Empty
Warning: partition 1 does not start at a cylinder boundary
Warning: partition 1 does not end at a cylinder boundary
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
  Physical volume "/dev/vdc1" successfully created
  Volume group "docker-vg" successfully created
  Rounding up size to full physical extent 16.00 MiB
  Logical volume "docker-poolmeta" created.
  Logical volume "docker-pool" created.
  WARNING: Converting logical volume docker-vg/docker-pool and docker-vg/docker-poolmeta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted docker-vg/docker-pool to thin pool.
  Logical volume "docker-pool" changed.

Enable and start docker service.

1
2
3
# systemctl enable docker
# systemctl start docker
# systemctl is-active docker

Ensure Host Access

On each hosts, generate an SSH key WITHOUT a password

1
# ssh-keygen

Copy the id_rsa.pub to each host:

1
2
3
4
5
# for host in master.example.com \
    node1.example.com \
    node2.example.com; \
    do ssh-copy-id -i ~/.ssh/id_rsa.pub $host; \
    done

Quick Installation

Running an Interactive Installation

Start the interactive installation by running under command, and follow the on-screen instructions to install a new OpenShift Continer Platform cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
$ atomic-openshift-installer install
*** Installation Summary ***

Hosts:
- master.example.com
  - OpenShift master
  - OpenShift node (Unscheduled)
  - Etcd
  - Storage
- node1.example.com
  - OpenShift node (Dedicated)
- node2.example.com
  - OpenShift node (Dedicated)

Total OpenShift masters: 1
Total OpenShift nodes: 3

NOTE: Add a total of 3 or more masters to perform an HA installation.

Gathering information from hosts...
All hosts in config are uninstalled. Proceeding with installation...

Wrote atomic-openshift-installer config: /root/.config/openshift/installer.cfg.yml
Wrote Ansible inventory: /root/.config/openshift/hosts

Ready to run installation process.

Play 1/28 (Create initial host groups for localhost)
..
Play 2/28 (Create initial host groups for all hosts)
.
Play 3/28 (Populate config host groups)
................
Play 4/28 (Ensure that all non-node hosts are accessible)
.
Play 5/28 (Initialize host facts)
................
Play 6/28 (Gather and set facts for node hosts)
...............
Play 7/28 (Verify compatible yum/subscription-manager combination)
..
Play 8/28 (Determine openshift_version to configure on first master)
............................................................................................
Play 9/28 (Set openshift_version for all hosts)
............................................................................................
Play 10/28 (Set oo_option facts)
........
Play 11/28 (Disable excluders)
..........................
Play 12/28 (Configure etcd)
................................................................................................................................................Pausing for 10 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
....................................................................
Play 13/28 (Configure nfs)
...............................................
Play 14/28 (Gather and set facts for master hosts)
.......................
Play 15/28 (Determine if session secrets must be generated)
..............
Play 16/28 (Generate master session secrets)
..............
Play 17/28 (Configure masters)
.............................................................................................................................................................................................................................................................................................................................................................................................................................................
Play 18/28 (Additional master configuration)
.......................................................................................................................................................................................................................
Play 19/28 (Gather and set facts for node hosts)
...............
Play 20/28 (Evaluate node groups)
..
Play 21/28 (Configure nodes)
.............................................................................................................................................................................................................................................................................................................................................
Play 22/28 (Additional node config)
.....................................................................................................................
Play 23/28 (Create persistent volumes)
............................................................................................................................................................
Play 24/28 (Create Hosted Resources)
.......................................................................................................................................................................................................Pausing for 30 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
................................................
Play 25/28 (Re-enable excluder if it was previously enabled)
...............
localhost                  : ok=11   changed=0    unreachable=0    failed=0   
master.example.com         : ok=606  changed=151  unreachable=0    failed=0   
node1.example.com          : ok=202  changed=39   unreachable=0    failed=0   
node2.example.com          : ok=202  changed=39   unreachable=0    failed=0   


Installation Complete: Note: Play count is only an estimate, some plays may have been skipped or dynamically added


The installation was successful!

If this is your first time installing please take a look at the Administrator
Guide for advanced options related to routing, storage, authentication, and
more:

http://docs.openshift.com/enterprise/latest/admin_guide/overview.html

Running an unattended Installation

Unattended installation allow you to run the installation with a pre-defined configuration file. The default installation configure file path is ~/.config/openshift/installer.cfg.yml. Define the configuration file and run the install command with the -u option.

1
$ atomic-openshift-installer -u install

Here is a simple example of the install.cfg.yml file. For further information, please follow Defining an Installation Configuration File

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
ansible_callback_facts_yaml: /root/.config/openshift/.ansible/callback_facts.yaml
ansible_inventory_path: /root/.config/openshift/hosts
ansible_log_path: /tmp/ansible.log
deployment:
  ansible_ssh_user: root
  hosts:
  - connect_to: master.example.com
    hostname: master
    ip: 10.64.221.200
    public_hostname: master
    public_ip: 10.64.221.200
    roles:
    - master
    - etcd
    - node
    - storage
  - connect_to: node1.example.com
    hostname: node1
    ip: 10.64.221.47
    node_labels: '{''region'': ''infra''}'
    public_hostname: node1
    public_ip: 10.64.221.47
    roles:
    - node
  - connect_to: node2.example.com
    hostname: node2
    ip: 10.64.221.192
    node_labels: '{''region'': ''infra''}'
    public_hostname: node2
    public_ip: 10.64.221.192
    roles:
    - node
  master_routingconfig_subdomain: ''
  openshift_master_cluster_hostname: None
  openshift_master_cluster_public_hostname: None
  proxy_exclude_hosts: ''
  proxy_http: ''
  proxy_https: ''
  roles:
    etcd: {}
    master: {}
    node: {}
    storage: {}
variant: openshift-enterprise
variant_version: '3.5'
version: v2

Also you can specify a different path of the configuration file with the -c option.

1
$ atomic-openshift-installer -u -c </path/to/file> install

Verifying the installation

After the installation is completed.

  1. Verify the master and nodes are started in Ready status. On the master host, run the following as root

    1
    2
    3
    4
    5
    6
    
    # oc get nodes
       
    NAME                        STATUS                     AGE
    master.example.com          Ready,SchedulingDisabled   165d
    node1.example.com           Ready                      165d
    node2.example.com           Ready                      165d
    
  2. The web console use the master host name with a default port number 8443. In this test environment, you can find the web console at https://master.openshift.com:8443/console

  3. Now that the install has been verified, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host:

    1
    
    # atomic-openshift-excluder exclude
    

Uninstallation

You can uninstall OpenShift Container Platform from all hosts using the follow commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ atomic-openshift-installer uninstall
OpenShift will be uninstalled from the following hosts:

  * master.example.com
  * node1.example.com
  * node2.example.com

Do you want to proceed? [y/N]: y

Play 1/9 (OSEv3:children)
....
Play 2/9 (nodes)
..
Play 3/9 (masters)
..
Play 4/9 (etcd)
..
Play 5/9 (nodes)
...............................
Play 6/9 (masters)
............
Play 7/9 (etcd)
............
master.example.com         : ok=60   changed=17   unreachable=0    failed=0
node1.example.com          : ok=35   changed=8    unreachable=0    failed=0
node2.example.com          : ok=35   changed=8    unreachable=0    failed=0

If you are using a configuration file, specify the file path for the uninstallation:

1
$ atomic-openshift-installer -c </path/to/file> uninstall