I developed thsi script to provide a gateway into a kubernetes cluster deployed on OpenStack. If I had been in public cloud, I would probably have used a managed load balancer. When I deployed the kubernetes cluster, I deployed the gateway server(s) as a nodes, so they would have the same network overlay (e.g. flannel, calico, etc.) as the rest of the cluster. This meant that HAProxy could route traffic directly to container workloads.
# Setup gateway
A gateway server is used to allow incoming traffic into the kubernetes cluster based on services. The steps below setup an HAProxy server for this purpose.
* Disable pod scheduling on the gateway node(s)
* Create ServiceAccount for gateway API calls
* Install HAProxy
* Copy over gateway config script and setup cron
* Some adjustment may be required to security groups (reference)
To run the `kubectl` commands below, first SSH into one of the master nodes through the bastion.
## Disable pod scheduling
The gateway is setup by kubespray as a node to facilitate pod access using the selected network overlay (e.g. calico). This first command disables pod scheduling so all resources are available to HAProxy.
```
kubectl cordon k8s-gateway-0
```
## Create ServiceAccount for gateway
The following commands _should be run on one of the master nodes_ (e.g. __k8s-master-0__) and will create a ServiceAccount, ClusterRole and a ClusterRoleBinding to tie them together. Even though the ServiceAccount has a namespace, ClusterRoles span all namespaces.
The commands below show details about the resources created above. The first two commands allow you to retrieve the TOKEN. The last command helps to verify that the TOKEN gives access to the expected resources. Obviously substitute the IP address for one of the master nodes in the cluster.
```
kubectl get sa gateway --namespace kube-system -o yaml
kubectl auth can-i get nodes --server=https://k8s-master:6443 --insecure-skip-tls-verify --token TOKEN
```
## Install HAProxy
Install HAProxy and python libraries using apt-get. The dynamic configuration of HAProxy requires jinja2, which is installed using pip below.
### Ubuntu or Debian
```
sudo apt-get update
sudo apt-get install -y haproxy python-setuptools
sudo easy_install pip
sudo pip install jinja2
```
### RHEL or CentOS
```
sudo yum install -y haproxy
sudo easy_install pip
sudo pip install jinja2
sudo mkdir /run/haproxy/
```
In some cases it can be necessary to validate haproxy. `haproxy -v` will print the current version of HAProxy installed. A configuration file can be validated using `haproxy -f /etc/haproxy/haproxy.cfg -c`. The HAProxy service can be managed using `sudo service haproxy start|stop|restart`.
## Create DNS A records
Create a single A record to resolve all traffic (wildcard) for the cluster. For example
```
A *.datacenter2.example.com 192.168.12.219
```
The IP above could be public, but it really just needs to be routable on the network that needs access to the workloads you run on kubernetes.
## Copy gateway config script and create cron job
Copy the following files from the `/eng_resources` directory
*`gateway-config-cron`
*`chmod +x gateway-config-cron`
*`gateway-haproxy-config.py`
*`haproxy.j2`
The last two files need to be in the same directory. The shell script `gateway-config-cron` needs to be updated with the correct path to `gateway-haproxy-config.py`.
The file `gateway-haproxy-config.py` needs to be updated with the IP address of one of the master nodes and the TOKEN for the ServiceAccount created above.
The file `haproxy.j2` needs to be updated based on the SSL certificate and domain name used in the next step.
`sudo crontab -e`
`* * * * * cd /home/centos && ./gateway-config-cron`
## Setup SSL automation
Kubernetes API interactions on port 7443 require SSL. This process uses [Let's Encrypt](https://letsencrypt.org/) to get a valid, signed certificate to provide SSL. This process also uses the [acme.sh script](https://github.com/Neilpang/acme.sh) to interact with Let's Encrypt.
The commands below will need to be adjusted to the cluster being configured. In the below example, the Austin Engineering cluster is assumed: `api.datacenter1.example.com`.
### Install acme.sh on the gateway
```
[root@k8s-gateway-0 ~]# curl https://get.acme.sh | sh
% Total % Received % Xferd Average Speed Time Time Time Current
[Thu May 3 06:15:11 UTC 2018] Your cert is in /root/.acme.sh/api.datacenter1.example.com/api.datacenter1.example.com.cer
[Thu May 3 06:15:11 UTC 2018] Your cert key is in /root/.acme.sh/api.datacenter1.example.com/api.datacenter1.example.com.key
[Thu May 3 06:15:12 UTC 2018] The intermediate CA cert is in /root/.acme.sh/api.datacenter1.example.com/ca.cer
[Thu May 3 06:15:12 UTC 2018] And the full chain certs is there: /root/.acme.sh/api.datacenter1.example.com/fullchain.cer
```
### Deploy the certificate
The following steps deploy the certificate and private key to HAProxy and then reloads. More details here: https://github.com/Neilpang/acme.sh/tree/master/deploy
[Fri May 4 16:02:12 UTC 2018] Full path to PEM /etc/haproxy/api.datacenter1.example.com.pem
[Fri May 4 16:02:12 UTC 2018] Certificate successfully deployed
[Fri May 4 16:02:12 UTC 2018] Run reload: /usr/sbin/service haproxy restart
Redirecting to /bin/systemctl restart haproxy.service
[Fri May 4 16:02:12 UTC 2018] Reload success!
[Fri May 4 16:02:12 UTC 2018] Success
```
### Automatic renewals
When acme.sh was installed (see above), it also installed a cron job. That cron job will monitor all certificates on the gateway (there's only one based on this process) and automatically renew them before they expire. This means you should never have to worry about monitoring the certificates or renewing them.
## Security groups and port assignments (reference)
The HEAT templates create and configure default security groups that expose web ports (80, 443), 7443 for `kubectl` API access and 8000-9000 for general use. Review the HEAT templates or the security groups for more details.
