csuttles (⌐■_■)

Memento Mori

As part of our efforts to improve the resilience of OpenStack within sites, we are moving to a multi controller architecture. I decided to tackle the underlying, stateful infrastructure services first. The first of those I looked into was RabbitMQ. RabbitMQ supports clustering, and HA queues, as well as durable queues (queues that persist to disk). The setup seems fairly simple, but unfortunately many things still assume IP means IPv4, and that's exactly where this got interesting.

In Mitaka release, I worked with our storage team and deployed a NetApp Cinder backend, using Fibre Channel connectivity from the nodes running cinder-volume. When I upgraded to Newton, we started seeing errors in the cinder-volume logs that seemed to match this bug: NetApp: Failed to get info for aggregate Unfortunately, at that time, there was not a fix available, we lost my POC for this project on the storage team, and the leaders in the org wanted to explore going a different way, so we dropped the cinder backend, instead of submitting a patch or finding a resolution.

We added availability zones to our deployment for a few reasons. Fault domain isolation The foremost of these was to isolate fault domains within our environment. Spreading workload across availability zones allows us to ensure that the applications and services provided by the OpenStack workload are resilient. Influencing placement Availability zones afford us more granular control of scheduling, without modifying configs or changing the default scheduler hints or filters. This allows us to be specific enough about placement to steer things in an intelligent way, while avoiding the burden of placing everything manually.

In my last post, I detailed a few things I experienced moving the Ocata release of OpenStack to an IPv6 only environment. While the major hurdles are complete, there are still more things to consider Image Automation We build images using Jenkins and Packer, and push those images into glance, which stores them in a Swift backend. That's all still going smooth, but where we ran into trouble is the automation of configuration.

Since I took the lead on the OpenStack deployment in my workplace, IPv6 has been a major goal of our deployment. IPv6 is a first class citizen in our infrastructure, and the other environments that OpenStack coexists with are already running IPv6 only. This made getting OpenStack to run IPv6 only a major milestone, which was finally accomplished during our move to Ocata release. Standards and SSL Termination While running client workload with IPv4 or dual stack is acceptable, or even required in some cases (vendor packaged VMs are notoriously guilty here), the standard for deploying infrastructure is high.

In my last post, I ran into some challenges with Ocata. The placement API and cells v2 configuration was not yet documented (that documentation is actually still only in draft form), and this made configuring them more difficult than most OpenStack services. This is where Devstack can be a very useful tool. In the post on ask.openstack where I found some help before there was much available, there's this very helpful suggestion for configuring cells and placement on devstack:

What's up, Doc? In Ocata release, the placement API and cells (v2) are mandatory. This is not currently documented in the official installation documentation: https://docs.openstack.org/ocata/install-guide-ubuntu/nova-controller-install.html edit: There's now a draft of docs, based on the work on the bug mentioned later in this post here: http://docs-draft.openstack.org/28/438328/12/check/gate-openstack-manuals-tox-doc-publish-checkbuild/846ac33//publish-docs/draft/install-guide-ubuntu/nova-controller-install.html At the time of writing this post, there is no mention of placement API of cells v2, which are both mandatory in Ocata release.

Everything begins with Keystone This is nothing new, but it's important to point out. Nothing works if Keystone is not set up properly. The easiest way to check if Keystone and your client are set up correctly is to run: openstack token issue An often overlooked part of Keystone is the service catalog. Keystone is responsible for authentication and authorization in OpenStack, but it's also responsible for the service catalog. If you run the following, you'll see the contents of the service catalog:

OpenStack is exciting! I started using OpenStack at work; in fact, my desire to work on it and learn more about it was a major factor in accepting my current position. The learning curve is steep, but don't let that intimidate you. The rewards for persistence are great. Crawl, Walk, Run OpenStack is a really large project, with a lot of moving parts. My approach, after some trial and error, and the approach I advise to people new to OpenStack is to crawl, walk, and run.