Use Traffic Control to Simulate Network Chaos in Bare metal & Kubernetes

This article explains how to use traffic control to simulate network chaos in Bare metal & Kubernetes, the network chaos can test the resilience of your service under a specified network condition such as packet loss, latency increase, etc.

What’s Linux traffic control?

Linux traffic control consists of shaping, scheduling, policing, dropping the traffic, it can be used to network administration, for example, rate limit the user traffic, setup the traffic priority.

How does traffic control work?

tc is the user-space utility program used to configure the Linux kernel packet scheduler, it supports rich functionality for the users to setup the traffic control.

Let’s start from a network chaos sample, suppose we have a news website which consists of a bunch of microservices and the news detail API from news service (which is used to render the news page) depends on the comment API from comment service, we want to test the application’s resilience when the comment API is down.

We want to use tc to block the egress traffic to the comment service, we’ll run it on the nodes of news service, the command will be

tc qdisc add dev eth0 root handle 1:0 prio
tc qdisc add dev eth0 parent 1:3 netem loss 100%
tc filter add dev eth0 parent 1:0 protocol ip prio 3 u32 match ip dst <comment-service-ip> flowid 1:3

To explain these commands, there are several components needed to be clarified,

• qdisc
  qdisc is short for ‘queueing discipline’ and it is elementary to understanding traffic control. Whenever the kernel needs to send a packet to an interface, it isenqueuedto the qdisc configured for that interface. Immediately afterwards, the kernel tries to get as many packets as possible from the qdisc, for giving them to the network adaptor driver.
• class
  Some qdiscs can contain classes, which contain further qdiscs - traffic may then be enqueued in any of the inner qdiscs, which are within the classes. When the kernel tries to dequeue a packet from such a classful qdisc it can come from any of the classes. A qdisc may for example prioritize certain kinds of traffic by trying to dequeue from certain classes before others.
• filter
  A filter is used by a classful qdisc to determine in which class a packet will be enqueued. Whenever traffic arrives at a class with subclasses, it needs to be classified. Various methods may be employed to do so, one of these are the filters. All filters attached to the class are called, until one of them returns with a verdict. If no verdict was made, other criteria may be available. This differs per qdisc.

The above commands can be interpreted as the tree below,

• ID
  Each node in the tree has a unique ID, it consists of two parts, “{major}:{minor}”.
  For the qdisc, its major part must be unique in the whole tree, its minor part is usually 0 and can be omitted in the tc command.
  For the class, its ID’s major part is the same as its parent major; its minor part should not be the same as its siblings.
• Root qdisc
  The first command tc qdisc add dev eth0 root handle 1:0 prio set priority qdisc for the root device eth0, as the name says, the packets are sent in prioritized order. Its ID is 1:0 which must be unique and 1:0 is reserved for the root qdisc.
• 3 classes
  The 3 classes are created by the prio qdisc by default, the packets are “routed” to one of them based on the priority. Their IDs are decided by the parent ID, in this case, major part must be the same as its parent 1, minor part is from 1 to 3.
• Leaf qdiscs
  The first 2 leaf qdiscs from left are fifo qdiscs which is set for prio qdisc by default. The third one is specified by the command and it overrides the default fifo qdisc, it uses netem qdisc to create 100% packet loss on class 1:3, so any packet sent to this class will be abandoned.
• Filter
  There’s only 1 filter in the tree, it routed the packets whose destination IP is comment service to the class 1:3, it will then abandon 100% of them.

As a result, these 3 commands will help create 100% packet loss from news service to comment service as network chaos.

How does tc cooperate with Docker?

Suppose I want to run the network chaos for a docker container like above, we have 2 ways:

Run tc within the target container

You can run tc within the docker container with NET_ADMIN capability, otherwise you’ll face Operation not permitted

docker run -it --rm --cap-add=NET_ADMIN <service-image> <start-command>

One more point is that the target container must have tc command, then you can use docker exec to modify the traffic control within the container.

However, containers are not always started with NET_ADMIN, so this method is limited in many cases.

Run tc on the host machine

Another way to modify it is to do it on the host machine where the container runs. As we know, the container’s network is isolated from the host machine by Linux namespace, so we can enter the container’s network namespace and modify the traffic control.

Here’s the workflow:

• First, we should get the docker container’s ID.

$ docker ps
CONTAINER ID   IMAGE              COMMAND       CREATED        STATUS        PORTS     NAMES
7d357060fe32   ubuntu:14:04       "/bin/bash"   5 weeks ago    Up 5 weeks              adoring_engelbart

• Second, extract the PID of the container.

$ docker inspect 7d357060fe32 | jq '.[].State.Pid'
14902

• Finally, modify the traffic control on the host.

# add prio qdisc to the root qdisc of the container
$ sudo nsenter -t 14902 -n tc qdisc add dev eth0 root handle 1:0 prio
# qdisc exists!
$ sudo nsenter -t 14902 -n tc qdisc show dev eth0
qdisc prio 1: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

How does tc cooperate with Kubernetes?

For Kubernetes, the mechanism is the same as docker one, the difference is that target containers are distributed in the cluster now, so you need to run the query first to locate the target containers and then run the tc (as we mentioned above) on the host machines where the target containers are running.

There are open source chaos testing solutions if you’re interested, like Litmus, Chaos Mesh, etc.

Conclusion

Traffic control is a powerful tool to manipulate the packets and it’s not a frequently used tool for most of us don’t need to administrate the network :) But since chaos testing is more and more adopted by the big companies and different chaos scenarios occur, it’s a good time to understand the technical details for more confidence on the tool.

Reference

• Traffic control Manual
• Linux Advanced Routing & Traffic Control HOWTO
• Linux Network Trac Control | Implementation Overview
• Traffic control Visualisation

Recommended Posts(Driven byHexo Recommended Posts plugin)

• How to set up a reasonable memory limit for Java applications in Kubernetes
• How to alert for Pod Restart & OOMKilled in Kubernetes
• Implement zero downtime HTTP service rollout on Kubernetes
• Practice datacenter failover in production