jcardena.com › Blog › Security and Ops from Day One
184 posts
EN ES

Security and Ops from Day One

Software

The boring parts of running production Kubernetes (secrets, least privilege, network isolation) are the real job. Security and operations are designed in, not bolted on.

Security and Ops from Day One

The first time something I ran got probed, it was not dramatic. No alarm, no movie hacker. Just a log line showing that a service I had cheerfully bound to the whole network, an internal model-serving endpoint as it happened, had been found by something that scans the whole internet looking for exactly that. Nothing was lost that time. But it taught me the lesson that has shaped how I build since: the demo takes an afternoon, and making it safe to leave running takes far longer. That part is the real engineering.

TL;DR: Security and operations are not a phase you add after launch. They are decisions you make in the first commit, whether you realize it or not. The interesting work in running real systems is the part most tutorials skip: how secrets flow, who is allowed to do what, and how much damage one compromised piece can do.

SecretsLeastPrivilegeNetworkIsolationSmall BlastRadius
Four decisions you make on day one, whether you mean to or not.

Default open is a decision you did not know you made

The most common security hole I find, including in my own work, is not a clever exploit. It is a default. A service that listens on every interface instead of just the one it needs. A port exposed to the whole network because that was the quickest way to make it reachable while testing. No rules about which pieces are allowed to talk to which other pieces, so the network is flat and anything can reach anything.

Individually each of those feels harmless. Together they describe a system where one compromised component owns everything. I have audited production clusters and found exactly this pattern: a service open to the world, no isolation rules, and in one embarrassing case an identity check that was effectively turned off because a token had been left blank. It worked perfectly, which is the trap. Working and safe are different tests, and only one of them shows up in a demo.

A maze of open gates glowing faint gold in a deep navy void
Default open is a decision you did not know you made: a flat network where anything can reach anything.

Secrets are not configuration

The fastest way to leak a credential is to treat it like ordinary config. Baked into an image, it travels everywhere that image goes and lives forever in its history. Sprinkled through environment variables across a dozen services, it becomes impossible to rotate because no one knows everywhere it lives.

The discipline is to treat secrets as their own category with their own rules. They are stored in a dedicated place built for them, handed to a workload only at the moment it runs, and never written into the thing you build and ship around. The test I hold myself to is simple: if I had to rotate every credential tomorrow because one leaked, could I do it in an hour without hunting? If the answer is no, my secrets are really just config wearing a disguise, and I have a problem I have not noticed yet.

Least privilege is a daily habit, not a setting

Every workload runs as some identity, and that identity can do some set of things. The lazy default is to give it broad permissions so nothing breaks. The habit worth building is the opposite: give each piece exactly what it needs to do its job and nothing more. It is more annoying up front and it pays for itself the first time something is compromised, because a narrow identity simply cannot reach much.

I think of it as drawing the smallest possible circle around each component. This model-serving route reads from that one vector store and calls that one inference endpoint; the agent's tool can write to that one queue and reach nothing else, and that is the entire list. When a request comes in to widen the circle, that is a real decision with a real reason, not a reflex. Least privilege is not a checkbox you tick once. It is a question you answer every time you add a moving part.

Blast radius is the question that matters

When I review a design now, the question I keep coming back to is not "can this be broken into," because given enough time and a determined attacker, assume yes. The better question is "if this one piece is owned, what can the attacker reach from there." That is the blast radius, and shrinking it is most of practical security.

Network isolation is how you shrink it. Instead of a flat network where everything can talk to everything, you write down which components are actually allowed to communicate, and you deny the rest by default. It takes more thought than leaving everything open. The payoff is that a compromise stays a small fire instead of becoming the whole building. The same thinking applies to how you split workloads, where you put boundaries, and what you let cross them.

Concentric navy walls and gold barriers containing a single contained burst of light
Shrink the blast radius: concentric barriers keep a compromise a small fire, not the whole building.

Operations is security wearing work clothes

People file security and reliability in separate drawers, but in practice they are the same drawer. A workload with no health checks and no resource limits is a reliability problem until the day it becomes a security one, when it exhausts a node and takes its neighbors down with it. A registry full of stale, unpatched images is an ops smell and a vulnerability at the same time. The cluster that runs out of disk and starts evicting things is having an outage that is also, quietly, a failure of operational hygiene.

The boring operational work, watching resource pressure, setting limits, keeping images current, making sure you can actually see what is happening, is not separate from keeping the system safe. It is the same work. Most of my real incidents have not been attacks. They have been operational failures that a little discipline up front would have prevented.

EXPOSURE SURFACEBind to OneInterfaceIngress atthe EdgeIDENTITYLeast-Privilege AcctsSECRETSVaultMounted atRuntimeRotatableNETWORKDefault-DenyNetworkPolicySmall BlastRadiusOPERATIONSResource LimitsHealth ChecksImage Hygiene
Defense in depth: every layer is a decision you make on day one.

The takeaway

I do not believe in a security phase anymore, the same way I do not believe in a testing phase. Both are how you build from the first line, or they are bolted on badly later after something has already gone wrong. Standing the service up was always the easy part. Everything after, the secrets and the identities and the isolation and the quiet operational habits, that is where the actual engineering lives. When I find a hole in my own infrastructure now, and I still do, I do not feel caught out. I feel like the process is working, because the cheapest place to find these things is before someone else does.

JC
Juan Cardena
Enterprise AI & Agentic Systems Architect

Enterprise architect with 25 years across web, software, data, and AI. MIT CDAO ’25. Writing on agentic AI in production.