We investigated the memory increase problem some time ago and learned a lot about JVM metrics. This happened again, we noticed several Java applications deployed in Kubernetes got the memory usage increasing gradually until it reached the memory limit, even after several times of increasing the memory limit, the usage can always hit above 90%, sometimes the container will be OOMKilled.

This article introduces some discovery of the Java memory usage in Kubernetes and how to set up a reasonable memory request/limit based on the Java heap requirement and the memory usage of the application.

This article introduces how to set up alerts for monitoring Kubernetes Pod restarts and more importantly, when the Pods are OOMKilled we can be notified.

This article explains how to use traffic control to simulate network chaos in Bare metal & Kubernetes, the network chaos can test the resilience of your service under a specified network condition such as packet loss, latency increase, etc.

You might have encountered some 5xx errors during http service rollout on Kubernetes and wonder how to make it more reliable without these errors, this article will first explain where this errors come from and how to fix them and implement zero downtime.

Prometheus is an opensource time series database, commonly used to gather and calculate monitoring metrics, this article explains how the query works with /query_range API.

This article is InfluxDB command cheatsheet about how to interact with influxDB server and query the metrics. The InfluxDB version I tested is v1.7.10

I’ll use an example to introduce how to build the smallest docker image to your best, a light image will accelerate image rollout and the fast build process will speed up your development cycle.

As a developer or operator of a Website, the certificate expiration could happen and make the services not work. I’ll introduce how to monitor certificates like SSL,JKS,P12 using Telegraf.

Certificates are broadly used for security reasons, they can be used within internal service or public service communication. The most common certificate is TLS used for verifying the identity of the HTTPS service. To increase security, the certificate will not be always valid because of expiration. To prevent the certificate expiry, we should rotate them periodically and meanwhile monitor them and alert if expired. Telegraf is a popular metric collecting tool to implement this.

Distributed system is like human body, it will have issues and break. There’s a theory that we feed it with issues deliberately and constantly, the body will be more and more stable and robust. It’s the same to system, put some issues to datacenters and let them failover automatically.