Kubernetes outgoing connections - database

Question

I am going to keep this simple and ask, is there a way to see which pod have an active connection with an endpoint like a database endpoint?
My cluster contains a few hundred of namespace and my database provider just told me that the maximum amount of connections is almost reached and I want to pinpoint the pod(s) that uses multiple connections to our database endpoint at the same time.
I can see from my database cluster that the connections come from my cluster node's IP... but it won't say which pods... and I have quite lot of pods...
Thanks for the help

Answer 1

Each container uses its own network namespace, so to check the network connection inside the container namespace you need to run command inside that namespace.
Luckily all containers in a Pod share the same network namespace, so you can add small sidecar container to the pod that print to the log open connections.
Alternatively, you can run netstat command inside the pod (if the pod has it on its filesystem):
kubectl get pods | grep Running | awk '{ print $1 }' | xargs -I % sh -c 'echo == Pod %; kubectl exec -ti % -- netstat -tunaple' >netstat.txt
# or
kubectl get pods | grep Running | awk '{ print $1 }' | xargs -I % sh -c 'echo == Pod %; kubectl exec -ti % -- netstat -tunaple | grep ESTABLISHED' >netstat.txt
After that you'll have a file on your disk (netstat.txt) with all information about connections in the pods.
The third way is most complex. You need to find the container ID using docker ps and run the following command to get PID
$ pid = "$(docker inspect -f '{{.State.Pid}}' "container_name | Uuid")"
Then, you need to create named namespace:
(you can use any name you want, or container_name/Uuid/Pod_Name as a replacement to namespace_name)
sudo mkdir -p /var/run/netns
sudo ln -sf /proc/$pid/ns/net "/var/run/netns/namespace_name"
Now you can run commands in that namespace:
sudo ip netns exec "namespace_name" netstat -tunaple | grep ESTABLISHED
You need to do that for each pod on each node. So, it might be useful to troubleshoot particular containers, but it needs some more automation for your task.
It might be helpful for you to install Istio to your cluster. It has several interesting features mentioned in this answer

Answer 2

The easiest way is to run netstat on all your Kubernetes nodes:
$ netstat -tunaple | grep ESTABLISHED | grep <ip address of db provider>
The last column is the PID/Program name column, and that's a program that is running in a container (with a different internal container PID) in your pod on that specific node. There are all kinds of different ways to find out which container/pod it is. For example,
# Loop through all containers on the node with
$ docker top <container-id>
Then after you find the container id, if you look through all your pods:
$ kubectl get pod <pod-id> -o=yaml
And you can find the status, for example:
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2018-11-09T23:01:36Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2018-11-09T23:01:38Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2018-11-09T23:01:38Z
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: 2018-11-09T23:01:36Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://f64425b3cd0da74a323440bcb03d8f2cd95d3d9b834f8ca5c43220eb5306005d