in DevOps, System Administration

The current state of OpenShift on Fedora 25

OpenShift has become the most wanted technology of Red Hat. 30% of their current sales are happening because of it. Today, we take a look at using the technology in it’s current state on Fedora 25. Is OpenShift usable without Red Hat support? And if so, what obstacles are there to overcome?

OpenShift 1.5 on Fedora Atomic Host 25

Our mission is setting up OpenShift on an OS with an up-to-date kernel. Additionally, we want rolling updates and as much support from Red Hat (community) as possible. In this setup, we therefore choose fedora atomic host (aka project atomic) over CoreOS. This helpful tutorial posted on project atomic in December 2016, the official documentation as well the the openshift-ansible docs are our guides.

The preferred setup is using the OpenShift ansible recipe. We are not happy with using the devel branch (3.6), so we are going to install the latest stable, which is 3.5 (Origin 1.5). Following the Readme we need to checkout the latest 3.5 tag. As not all tags are listed on github we identify the right one using git tag | sort.

openshift-ansible-3.5.69-1
openshift-ansible-3.5.7-1
openshift-ansible-3.5.70-1
openshift-ansible-3.5.71-1
openshift-ansible-3.5.8-1
openshift-ansible-3.5.9-1
openshift-ansible-3.6.1-1
openshift-ansible-3.6.10-1

openshift-ansible-3.5.71-1 seems to be the latest, so we are going to use this.

Next, we need an inventory file as stated in this project atomic post. Our config is the same as in this link and yes, we use containerized installation as this is the only available installation method for atomic host.

After setting everything up we run the playbook. And it fails. An issue on github refers to what we encountered and the bad news is: No fix for 3.5 .

Verdict: OpenShift 1.5 on Fedora Atomic Host 25

Our goal was to run OpenShift in a way to allow running it in production with an up-to-date kernel on an OS with a rolling release strategy. The first thing to do when figuring out if something is production ready is trying to install it in the preferred way (according to the docs). My colleagues and I were not able to install OpenShift on Fedora Atomic Host 25 with the official ansible playbook. Next steps would be to manually tamper with the ansible roles, but as the openshift-ansible repo has more than 200.000 lines of code, there really should be a working playbook to use without the need of manual labor for installing OpenShift on Fedora.

To get a complete picture we tried installing the devel version OpenShift 1.6 on Fedora Atomic Host 25 which failed as well. I give you more information about that step below when discussing installing OpenShift on Fedora 25.

OpenShift 1.5 on Fedora 25

Redefining our previous goals to make installation somehow more achievable we skip the self-imposed goal “rolling updates” and stick to Fedora 25. Project Atomic is known for being not the most loved kid around the block, so maybe using a rpmtree distro is not desirable in terms of support.

Steps are the same as above but we do not use containerized hosts as we want to stick to the best support possible, which means using the default / recommended way:

The default method for installing OpenShift Origin on Fedora, CentOS, or RHEL uses RPMs. – Installing on containerized hosts

Let’s take a look of what we are trying to achieve now:

  • Install OpenShift 3.5 on
  • Fedora 25 using the
  • latest openshift-ansible recipe with the
  • RPM package method

We stick to the aforementioned inventory file once again, using the latest openshift-ansible 3.5 tag and disabling the containerization method: containerized=false.

First try fails.

Using the recipe with python2 is not supported on Fedora, so we need yaml for python3. Unfortunately, the recipe for this task seems to be broken:

> ansible-playbook -i myinventory playbooks/byo/config.yml -e 'ansible_python_interpreter=/usr/bin/python3'

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: No module named 'yaml'
fatal: [myserver]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_o2dr5u0q/ansible_module_openshift_facts.py\", line 17, in <module>\n    import yaml\nImportError: No module named 'yaml'\n",
    "module_stdout": "",
    "rc": 1
}

The pyyaml package is missing although it should be installed by the openshift role. That means we are forced to install the package manually running ansible -i myserver -m command -a "dnf -y install python3-PyYAML" workers. Afterwards, we run the ansible playbook once again, leading to following output:

MSG:

MODULE FAILURE

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: RuntimeError: dictionary changed size during iteration
fatal: [myserver]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_dqlk4vga/ansible_module_openshift_facts.py\", line 2491, in <module>\n    main()\n  File \"/tmp/ansible_dqlk4vga/ansible_module_openshift_facts.py\", line 2478, in main\n    protected_facts_to_overwrite)\n  File \"/tmp/ansible_dqlk4vga/ansible_module_openshift_facts.py\", line 1899, in __init__\n    for (k, v) in self.system_facts.items():\nRuntimeError: dictionary changed size during iteration\n",
    "module_stdout": "",
    "rc": 1
}

I did not go into detail above but yes, it is the same error as before.

As this bug has been fixed on devel version OpenShift 1.6, we will try devel again:

OpenShift 1.6 on Fedora 25

Version 1.6 is devel, so let’s prepare ourselves for a bumpy road. However, it is not working at the moment, so what else could go wrong?

Once again we look for the latest release running git tag | sort. We choose openshift-ansible-3.6.68-1 which happens to be the release our nasty bug was fixed in. After checking out the release, we adapt our configuration openshift_release=v1.6.0 and run the playbook one more time:

fatal: [myserver]: FAILED! => {
    "changed": false,
    "failed": true
}

MSG:

{u'package_found': False, u'returncode': 1, u'stdout': u'', u'cmd': u'/usr/bin/repoquery --plugins --quiet --pkgnarrow=repos --queryformat=%{version}|%{release}|%{arch}|%{repo}|%{version}-%{release} --config=/tmp/tmpej3wsfer origin', u'results': {}, u'stderr': u'Traceback (most recent call last):\n  File "/usr/bin/repoquery", line 34, in <module>\n    import yum\n  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 59, in <module>\n    import config\n  File "/usr/lib/python2.7/site-packages/yum/config.py", line 30, in <module>\n    from parser import ConfigPreProcessor, varReplace\n  File "/usr/lib/python2.7/site-packages/yum/parser.py", line 4, in <module>\n    import urlgrabber\n  File "/usr/lib/python2.7/site-packages/urlgrabber/__init__.py", line 55, in <module>\n    from grabber import urlgrab, urlopen, urlread\n  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 512, in <module>\n    import pycurl\nImportError: pycurl: libcurl link-time version (7.50.3) is older than compile-time version (7.51.0)\n'}

Alright, this does not seem too bad, let’s fix it by executing ansible -i myserver -m command -a "dnf -y install libcurl-devel" workers

Hopes are high, let’s run the playbook again:

TASK [openshift_version : Get available origin version] *******************************************************************************************************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: a bytes-like object is required, not 'str'
fatal: [myserver]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_8bjp57qb/ansible_module_repoquery.py\", line 635, in <module>\n    main()\n  File \"/tmp/ansible_8bjp57qb/ansible_module_repoquery.py\", line 626, in main\n    rval = Repoquery.run_ansible(module.params, module.check_mode)\n  File \"/tmp/ansible_8bjp57qb/ansible_module_repoquery.py\", line 586, in run_ansible\n    results = repoquery.repoquery()\n  File \"/tmp/ansible_8bjp57qb/ansible_module_repoquery.py\", line 549, in repoquery\n    processed_versions = Repoquery.process_versions(rval['results'].strip())\n  File \"/tmp/ansible_8bjp57qb/ansible_module_repoquery.py\", line 468, in process_versions\n    for version in query_output.split('\\n'):\nTypeError: a bytes-like object is required, not 'str'\n",
    "module_stdout": "",
    "rc": 1
}

MSG:

MODULE FAILURE

This is where we need to stop our adventure of installing OpenShift on Fedora 25. More information about this bug can be found on github.

Switching to containerization leads to the same error.

Conclusion: OpenShift on Fedora 25

The installer is currently broken. That’s what we did:

  • We chose OpenShift 1.5
  • We chose stable Fedora 25 and stable Fedora Atomic Host 25
  • We used the recommended way of installing OpenShift using openshift-ansible

Errors we encountered:

  • The dependency installation procedure is broken (missing to install python3-pyyaml).
  • We encountered python type conversion errors and
  • python dictionary out of range errors.

Conclusion

Cloud computing is a very lively field of expertise. Several companies claim to have the best possible cloud solution. OpenShift is a highly opinionated kubernetes stack with a high level of complexity when it comes to bringing and holding together all moving parts.

If we take the current state of the openshift-ansible repository in conjunction with Fedora 25, it seems as if OpenShift on Fedora 25 is currently not production ready. This does not necessarily mean that you can’t have a running OpenShift cluster using Fedora 25 in production. It just means that you won’t be able to achieve this goal with the current install procedure and are very likely forced to do a lot of configuring and installing tasks yourself, leading to possible pitfalls during upgrade procedures eventually.