How to survive in an air gap datacenter
Recently I had an opportunity to expand my DevOps experience into Machine Learning Operation (MLOps). It was exciting as I have to install an enterprise grade software in an Air Gap Datacenter.
Unfortunately due to inexperience, I have to go through many difficult issues when coming to install software in an air gap environment. Due to zero internet access policy (no proxy no satellite servers), it is difficult for someone who is so comfortable with the cloud, who can practically yum install or apt installanything and everything. But I was eventually able to overcome the necessary hurdles to finally install and configure the software. Hence I hope by sharing my experience it helps others.
So these are my tips to make your DC trip easier if your DC is air-gapped:
Bring a jacket
You will be spending more time in an air gapped datacenter than one that is not. So bring a jacket to keep yourself warm.
Know the terrain of the DC
Know the rules, the dos and the don’ts, where is the pantry, toilet, desks where you have internet access before entering the air gap datacenter. It helps a lot.
Also there are some strict rules like handling of power after 7pm so take note of that.
Prepare local yum/apt repository
You definitely need a lot of libraries cos the things you install may have a lot of dependencies., Hence you definitely need a local repository server to help you in your libraries installation. I was using Red Hat as the base OS so there are useful links to help you get the full range of EPEL dependencies.
For libraries that are outside EPEL, use yumdownloader to download.
Work with vendors
If you work in a System/Software Integrator (SI) company, you will have to work with multiple parties. I requested the vendor who help setup the servers and network in the rack to help me install 3 libraries in all the redhat OS: net-tools, rsync and createrepo. These 3 are essential to help you set up your local repository. Also I ask them how do I copy those libraries into the server and I learned that I can use iLo which is something similar to KVM technology that I used to work on.
Get the server disk storage allocation right
Enterprise software has very strict disk requirements. Certain folders must have certain amount of empty space if not the whole installation does not work. Make sure / and /var have sufficient space for installation.
Get a power converter plug for your laptop
This is important. Your battery will run flat in no time when trying to rsyncing all the EPEL repository files into your server. Get a C19/C20 power plug converter for your laptop so it won’t run out of power
