Accessing a custom Flask microservice from a Kubernetes ingress controller

In this post, I’ll show you how to set up an ingress controller on a local kubernetes cluster to access a service that exposes a local flask server. The post is not meant to cover kubernetes concepts such as pods, services or ingress controllers or writing and containerizing a simple flask server. There is a lot of information about these concepts in kubernetes documentation and various blog posts. In this post, I’ll focus on some tips on how to troubleshoot when you try to access your flask server endpoint through a ingress controller and get a 404 error.

Note that there are two nginx-kubernetes ingress controllers:

This doc outlines the differences. I’m using the first one that seems far more popular.

The scripts and yaml files used in this post are in this repo. We’ll first set up a single node kubernetes cluster on an Ubuntu workstation, and then configure some simple pods and services which we’ll try to access via an ingress controller. Then, we’ll do the same with a simple flask server. Along the way, I’ll show you some troubleshooting steps to help debug 404 messages from the ingress controller and the flask server.

Setting up a local kubernetes cluster

Here’s a script to set up a single node kubernetes cluster on an Ubuntu workstation, using Kubeadm.

You can ignore the part that sets a custom setting for the horizontal pod autoscaler (hpa). That is not relevant to this post. Note that we set up the nginx ingress controller over a NodePort service, which is the simplest and easiest to understand option, specially if you are not using a cloud provider. See this  installation guide for details.

Accessing some simple services through the ingress controller

After completing the set up shown above, verify that ingress controller service is running in the ingress-nginx namespace. As shown in the screenshot below, you can access this service by curling the IP address of the ingress-nginx-controller service or using the node port. Doing either should show a “404 Not Found” page. If this doesn’t work, there is some issue with your set up and none of what follows will work either.

If you are curious about the configuration of the nginx server backing the nginx ingress controller, you can exec into the pod corresponding to the ingress-nginx-controller, as shown below. See this for more info.

Now, we’ll set up a couple of pods and expose them through services. Then, we’ll create an ingress resource to specify path based routing to those services. The screenshots below show this in action. See ingress.yaml in the github repo for configuration details.

All of this should work out of the box.

Creating a containerized flask server

Now, we’ll create a very simple flask server that exposes a couple of endpoints and create an image that runs it. We will set up a custom 404 error message, so we can distinguish our server 404 responses from those of nginx. Then, we’ll create a kubernetes pod using the server image and a service that exposes the server.

Next, we create a docker image and a pod that runs that image. We’ll also expose the pod through a service and verify that we can curl it. See the Dockerfile and server_pod.yaml in the github repo for this post.

Next, we’ll add a route for our service in the ingress. This is done by adding the following section to the ingress.yaml. This is already included in the ingress.yaml included in the github repo.

We intend traffic directed at 127.0.0.1/31338/my-server to be sent to my-server-svc service. Kubernetes DNS will automatically look up the service IP address for you. If the service is in a different namespace, you’ll need to use the fully qualified name of the service.

Delete and recreate the ingress resources and curl the endpoint. You can use kubectl delete -f ingress.yaml –force to delete all resources specified in the yaml. We get a 404 message, but we can tell that it is coming from our server.

If we run kubectl describe ingress on our ingress resource, we can see that it added a configuration for our service and was able to look up the correct service IP address.

If you delete and recreate the my-server-svc service, kubernetes will assign it a new IP. However, the service IP in the ingress configuration will update automatically. This is an important benefit of using ingress – you can access your services using a hostname or path, rather than worry about dynamic IP addresses.

Next, lets look at the logs of the ingress-nginx-controller pod. You can get the name of the pod running the ingress controller by using the kubectl get pods -n ingress-nginx command.

The logs give us a clue about why we are getting a 404 error from our server. Our server expects to receive GET requests on “/” and “/mars”, but the ingress controller is appending the path prefix. To solve this, we need to rewrite /my-server to /, /my-server/mars to /mars etc. This can be accomplished using “rewrite annotations“. Modify the ingress rule as follows (see ingress_final.yaml).

Using this annotation, any characters captured by (.*) will be assigned to the placeholder $2, which is then used as a parameter in the rewrite-target annotation. See this for more info about rewrite rules.

Delete and recreate the ingress. Now, viola! everything works 🙂

Hope you found this information helpful. Please leave a comment if you did!

1 Comment

  1. Thanks, Ankur, tried it; worked fine for me.
    I have been doing a lot of work w Tornado and Flask micro-services, while working on Athena, JP’s Python based eco-system for trading and risk pipelines, so I am very much familiar with the setup, but not what you described here.

Leave a Reply

Your email address will not be published.


*