Kubernetes is not just a new way to deploy. It changes your operational habits

When applications run as manual deployments on specific servers, all of a team's operational knowledge is built around those servers: which machine runs what, where the logs are, what "normal" load looks like. Moving the same platform to Kubernetes, we quickly learned that building images and manifests is the smaller part of the job. The bigger part is changing habits — in monitoring, in logging and in the application architecture itself.

Here are four areas where the old approach stopped working, and what replaced it.

Monitoring: from hosts to services

On servers, Zabbix-class monitoring works very well: a host, its services, disks, ports, processes. The problem in Kubernetes is not that Zabbix "misses" things. It is that it cannot see inside the cluster. Pods talk over the cluster's own network (an overlay), have addresses that live only as long as they do, and are not fixed targets you can list in a monitoring config. Zabbix sees what is on the surface: nodes, their load, the availability of exposed services.

Prometheus and Grafana give us the view inside:

metrics scraped from pods automatically, with labels (application, environment, version), rather than from a list of hosts,
alerts defined at the service level — "the API error rate crossed a threshold" — rather than "the process on machine X is not responding",
container resource views: memory usage against the limit, CPU throttling, restarts.

That last point turned out to be crucial. Sporadic 503s in one service had nothing to do with the code — the container was being throttled by the memory-limit mechanism. On a server that would never have happened, and classic host monitoring could not show it.

Zabbix did not go away — it still watches the physical nodes, the network and services outside the cluster. But what happens inside, we now look at with different tools.

Logs: a pod disappears together with its logs

On a server, grep in /var/log or journalctl -u service solves most problems. In Kubernetes both habits fail for one reason: pods are ephemeral. A new deployment, a scale-down or a restart after hitting the memory limit, and the previous pod's logs are gone. Logs of running containers rotate too — for us, the history on a node sometimes reached back only a few hours.

On top of that, one service is several replicas on different nodes. Finding an error would mean logging into each node and guessing which pod handled the request.

The answer is a log aggregator — for us, Loki with Grafana:

logs from every pod land in one place and stay there after the pod is gone,
search by labels (service, environment, pod) instead of by server name,
logs and metrics in the same tool — from an error chart you can jump straight to the logs from the same time window.

The habit change is easy to describe and hard to adopt: in an incident, the first reflex must be to open the aggregator, not a terminal on a node.

Horizontal scaling: a foot-gun if the application is not ready for it

One line in a manifest — replicas: 3 — and the service runs as three instances. It is one of Kubernetes' biggest promises and also a trap. An application written on the assumption that it runs once does everything several times after scaling out.

We learned this with customer notifications. After increasing the number of replicas, messages that were meant to go out once a day started going out twice. Fortunately not at the same moment — several hours apart, because each replica had its own task schedule. Nothing broke in the code; only the number of instances running that code changed.

We fixed it in two ways, depending on the kind of task:

a database lock — before starting, the task takes a lock in the shared database; if another replica already holds it, the run is skipped,
leader election — one leader is elected among the replicas and is the only one that runs periodic tasks; when the leader goes away, another replica takes over.

The same principle applies to everything the application keeps locally: in-memory locks, caches, sessions, temporary files. Anything that is supposed to be shared between instances must really be shared — in the database, in a queue, in external storage.

Native cluster tools

Sometimes it is better not to run a scheduler inside the application at all. Kubernetes has its own mechanism for this: the CronJob. A periodic task becomes a separate, short-lived pod started by the cluster — exactly once, with its own run history, resource limits and retry policy. The application serving traffic then needs to know nothing about schedules.

It is not always that simple. Tasks often rely on the application's context — configuration, connections, domain logic — and extracting them takes refactoring. So in practice we combine both approaches: new periodic tasks are designed as CronJobs from the start, while existing ones are protected with a lock or leader election until it is time to extract them.

Locality: the scheduler does not know what is close

On servers, a person decided what ran next to what: the database and the application that hits it hardest sat on the same machine or in the same rack. In Kubernetes that decision belongs to the scheduler — and without extra hints it looks mainly at free resources on the nodes, not at how far apart cooperating components end up.

Our cluster spans several physical servers, and the nodes talk over an overlay network. A pod that lands on a node on a different physical server than the database it uses pays for every query with an extra trip through the tunnel and the network between servers. Individually that is fractions of a millisecond; with hundreds of queries per user request it is a noticeably slower application, with no change to the code. Such a slowdown is easy to miss, because it comes and goes depending on where the pods happened to be placed after the last deployment.

Kubernetes has the tools for this, but you have to use them deliberately:

topology labels on nodes — which physical server (or zone) a node runs on; without them the scheduler has nothing to infer distance from,
pod affinity — "run this component close to that one", e.g. an application next to its database or cache,
pod anti-affinity and topology spread constraints — the opposite: replicas of the same service spread across physical servers, so that one machine failing does not take them all down at once,
local storage — a volume on a specific node's disk ties the pod to that node; it is fast, but it limits the scheduler's freedom and has to be part of the failure plan.

Locality and resilience pull in opposite directions — the closer components are, the more likely they fail together. There is no good "default" setting; it has to be designed for each critical path separately.

A pre-migration checklist

Before you move an application from a server into a cluster, check:

Monitoring — does the application expose metrics (e.g. for Prometheus), and are alerts defined at the service level rather than the host level?
Logs — does the application write to standard output, and do logs reach an aggregator that survives a pod restart?
Periodic tasks — what happens when the application runs as three instances? Lock, leader or CronJob?
Local state — sessions, caches, temporary files, in-memory locks: does anything assume there is only one instance?
Resources — are memory and CPU limits sized to real usage, and does monitoring show throttling?
Configuration — can everything that lives in files on the server today be passed in through environment variables and secrets?
Locality — which components must run close together (affinity) and which far apart (anti-affinity, replica spread)? Do the nodes carry topology labels?

Kubernetes gives you rolling updates, automatic restarts and scaling with a single command. But only a team that has changed how it looks at a running application — from "that one on that server" to "a service whose instances come and go" — actually benefits from them.

More on the infrastructure itself: Moving a dozen-plus applications to our own k3s cluster.