What is Openstack?
OpenStack is a platform used to build and manage cloud environments, ideally for private clouds, often using bare-metal servers. It offers several services that are onpar with hyperscalers like AWS, such as Nova (EC2), Cinder (EBS), Swift (S3), Neutron(VPC) and Keystone(IAM). Our main usecase was provisioning VMs and Block storage.
Task
I was asked to deploy OpenStack on a bare-metal Ubuntu server with specs I had only heard of until then. The server had 175 GB of memory and 42 CPU cores, making it a beast of a machine. I stress-tested it, checked htop and saw the bars hitting their maximum. The machine seemed perfect for our requirements, so I dove into the task.
Configuration
Before starting the deployment, I set the stage by configuring passwords for everything, following the password rules and storing them in environment variables, as shown in the figure. The setup seemed straightforward until I began encountering unexpected errors.
Problem
At first, the problems seemed straightforward, mainly dependency issues. However, an error message saying DBConnectionError started popping up, reading Can't connect to MySQL server at "@127.0.0.1". It retried 10 times before failing completely.
I checked the host system logs but found nothing. I also checked for port number conflicts, but everything seemed fine. I completely restarted the setup from scratch after failing to figure out what went wrong, but still faced the same issue. After a couple of hours of debugging and staring at the screen, I finally understood that the error indicated that the system was trying to connect to "@127.0.0.1" instead of just "127.0.0.1"
Issue
In connection strings especially in database URLs, "@" is used to separate the password from the hostname.
For example: mysql://username:password@hostname
If the password ends with "@", the parser might get confused, interpreting it as the start of the hostname, which leads to connection errors, which is exactly what happend in this case.
Takeaway
After figuring out that the "@" in the password was causing the issue, I fixed it by updating the password. Once I did that, the deployment worked fine, and OpenStack was up and running as expected.
It was a good reminder that sometimes the smallest details, like a special character in a password, can lead to big problems. But with a little patience and some debugging, I got through it.