In a MongoDB sharded cluster, it is recommended to run mongos locally within the application to avoid network overhead that can slow down database queries. But how does this fit into a decentralized architecture that most enterprises are shifting towards and more importantly, how does this recommended model fit within a containerized infrastructure? Well, it doesn't really, which is what we are going to be discussing today.
Container architectures do not lend themselves well to the recommended and standardized way of deploying mongos. For starters, mongos is not cgroups aware, which means it can blow up your CPU usage creating tons of TaskExecutor threads. Secondly, grouping containers in Kubernetes Pods ends up creating tons of mongos processes, resulting in additional overhead. This blog post will address some of the challenges associated with implementing mongos in a containerized environment and tips on circumventing these challenges.
Performance Impact with Containerized Mongos
If you run mongos in a container, you need to be aware that it won’t perform the same way as mongos in a non-containerized environment. Mongos with default sharding parameters combined with cgroup unawareness has the potential to affect the performance of other containers in the same host.
Being cgroups unaware means that the mongos process has no way of knowing the number of cores assigned to the actual process. Mongos is only aware of the number of cores assigned to the host itself. The lack of visibility into the assignment of cores per process results in mongos spawning the wrong number of threads and stealing CPU from other containers on the host. Let’s understand this with an example.
NGNIX is an open source software for web serving and caching and just like mongos, is not cgroups aware. If you deploy NGINX as a container on a host with 24 cores but only assign four cores to NGINX itself, it will still recognize all of those 24 cores and spawn the same number of worker threads. Mongos works the same way while spawning its own TaskExecutor connection pool, by overlooking the number of cores assigned to the process and blindly associating with the total number of cores that exist on the host. This characteristic can result in CPU starvation and affect other containers running on the same host.
How to Avoid TaskExecutor Connection Pool Explosion?
So, how do we circumvent this condition? To avoid pool explosion, set the parameter taskExecutorPoolSize in your containerized mongos by running it with the following argument, or setting this parameter in a configuration file: --setParameter taskExecutorPoolSize=X, where X the number of CPU cores you assign to the container (for example 'CPU limits' in Kubernetes or 'cpuset/cpus' in Docker).
Be aware that there are a minimum of four TaskExecutor connection pools. If you want to avoid CPU starvation and not affect the performance of other containers in the same host, you should assign a minimum of 4 CPU cores to your mongos container.
How do you know how many TaskExecutor pools your mongos is creating? Check the mongos logs; you will see entries like the one below:
“I ASIO [NetworkInterfaceASIO-TaskExecutorPool-X-0]”,
where X is the index of the TaskExecutor starting with 0.
Kubernetes Pods with Mongos Containers
In a Kubernetes environment, it is recommended that you run a mongos container per Kubernetes Pod, to simulate the “localhost” feeling for other containers. However, this is not optimal for performance. Each time a mongos process starts, it fetches all the cluster metadata from the mongo config servers, including the chunks metadata. Depending on the size of the cluster, these chunks can get very big. For instance, our deployment would generate millions of chunks and fetching vast amounts can be detrimental to performance. Given the dynamic nature of containers, especially with Kubernetes, mongos will restart more frequently than in a non containerized system, thereby creating a heavy load on the mongo config servers. If you have multiple mongos containers running in Kubernetes and a rolling upgrade restarts a bunch of them at once, your mongo config servers won't be able to keep up. They will struggle with the reads and mongos will take longer to be active and ready. As you can imagine, this is not good. Your Kubernetes Pods and your mongos instance must be available as soon as possible.
How to Avoid Overloading with Kubernetes
Instead of running a mongos container per Pod, you can simply run a mongos process per Kubernetes host and instruct your containers to use the centralized mongos instance instead of the 'localhost' one. At the end of the day, although they are not using the localhost connection, they will still reach mongos locally. But how will the containers know where to look for mongos?
Kubernetes has what is called the 'Downward API' that can show some host properties to the containers, and one of them is the IP of the host (status.hostIP, added in Kubernetes version 1.7). Your application just needs to know the host IP address of the mongos instance. You can then run mongos either with a DaemonSet or by running one in each host. Remember that you don't need a Kubernetes Service for your mongos process if you run it as a DaemonSet.
This is an example of a Kubernetes manifest using the 'Downward API':
apiVersion: v1
kind: Pod
metadata:
name: downward-api-for-mongos
spec:
containers:
- name: my-app-that-uses-mongos
image: k8s.gcr.io/myapp:1.24
env:
- name: HOST_IP_PATH
value: '/etc/podinfo/host_ip'
volumeMounts:
- name: podinfo
mountPath: /etc/podinfo
readOnly: false
volumes:
- name: podinfo
downwardAPI:
items:
- path: "host_ip"
fieldRef:
fieldPath: status.hostIP
Your app just needs to load the content of the file that HOST_IP_PATH env variable has to find the host IP and starts the connection to mongos.
Conclusion
The official documentation for mongos in containers is limited, and it is hard to find clarity about this on the internet. There is no guarantee that tools that are proven to work efficiently in a non-containerized environment will do in a containerized environment as well. Understanding how they work and how resources are utilized is critical before architecting and migrating legacy infrastructure to containers.