Everyone tells you EKS is “just Kubernetes.” Nobody tells you about the production surprises waiting on the other side.

Running Kubernetes in a tutorial is one thing. Running it in production on AWS is a completely different game.

I’ve spent months building a real-world DevOps course with 55+ hands-on demos, and along the way I hit every production gotcha you can imagine. Here are the five lessons I wish someone had told me upfront each one could save you hours of debugging and hundreds of dollars in wasted cloud spend.

This one surprised me the most.

Cluster Autoscaler (the default scaling solution most teams start with) only removes completely empty nodes. If a node is running at 10% utilization? It stays. If you have four nodes each running a single small pod? All four stay running. You keep paying for all of them.

Here's what happened in my demo:

I deployed 10 pods across 4 nodes, then scaled the deployment down to 2 pods. With Cluster Autoscaler, all 4 nodes stayed running — because each still had at least one pod on it.

Then I switched to Karpenter.

Within 30 seconds, Karpenter analyzed the remaining pods, consolidated them onto a single right-sized node, and terminated the other three. Four nodes became one. That's a 75% cost reduction just from intelligent bin-packing.

Why Karpenter is different: Karpenter doesn't just remove empty nodes — it actively looks for opportunities to consolidate workloads onto fewer, better-sized instances. It considers pod resource requests, topology constraints, and instance type availability to find the optimal placement.

What this means for you: If you're running EKS with Cluster Autoscaler, you're likely paying for nodes that are barely utilized. Karpenter's consolidation feature alone can cut your compute costs dramatically.

The course covers both OnDemand and Spot NodePools with live consolidation demos in Section 17.

Spot instances offer up to 70% savings over On-Demand pricing. That's a massive cost reduction. But there's a catch that most tutorials gloss over: AWS can reclaim your Spot instances with just a 2-minute warning.

Without proper handling, here's what happens:

AWS decides it needs the capacity back. Your instance gets terminated. Your pods die immediately. Requests in progress fail. Users see errors. Your PagerDuty goes off at 3 AM.

With proper handling (Karpenter + EventBridge + SQS), here's the flow:

The result: 70% cost savings with zero downtime. Your users never know it happened.

I built a complete demo where I simulate Spot interruptions and show the entire flow in real-time. It's one of the most eye-opening sections in the course (Section 17), because you can actually watch Karpenter orchestrate the replacement before the old node dies.

We've all done it. A quick env variable in a YAML manifest. A database password in a ConfigMap. "I'll fix it later."

Later never comes. And then:

The production-grade approach:

AWS Secrets Manager + Secrets Store CSI Driver. Here's how it works:

What this means in practice: Your security team is happy (secrets are centralized and audited). Your ops team is happy (no restarts during rotation). Your developers are happy (they just read from a file path).

Section 9 in the course walks through the complete setup with AWS Secrets Manager integration, Secrets Store CSI Driver, and Kubernetes manifests that reference them.

When people say "add monitoring," they usually mean install Prometheus and Grafana and call it done. But production observability has three distinct pillars, and each requires its own pipeline:

Traces — Follow a single request across multiple microservices
→ Instrumented with OpenTelemetry → Collected by ADOT Collector → Sent to AWS X-Ray

Logs — Capture application output and errors
→ Collected by a dedicated ADOT Collector → Sent to Amazon CloudWatch Logs

Metrics — Track resource usage, request rates, error rates
→ Collected by a third ADOT Collector → Sent to Amazon Managed Prometheus → Visualized in Amazon Managed Grafana

Three pillars. Three ADOT collectors. Three AWS destinations.

The cost trap nobody warns you about:

Kubernetes runs health check probes (liveness and readiness) every 10 seconds on every pod. Each probe generates a trace. With 20 pods, that's 120 traces per minute — 172,800 traces per day — just from health checks. Your X-Ray bill explodes with completely useless data.

The fix: Configure the OpenTelemetry Collector to filter out health check traces (/health, /ready endpoints) before they reach X-Ray.

I reduced my tracing costs by 85% with this single configuration change. Same visibility into real user requests. 85% lower cost.

Section 20 covers the complete ADOT setup with Amazon Managed Prometheus (AMP), Amazon Managed Grafana (AMG), and X-Ray integration.

Every Kubernetes tutorial uses SQLite or an in-memory database. It works great for demos. Then you get to production and realize:

Five microservices. Five different AWS managed services. One application.

In the course, students build a complete retail store application with this real-world data plane:

Each service has its own connection pooling, retry logic, and secrets management. Everything is provisioned with Terraform — one terraform apply creates the entire data layer.

Why managed services over running databases in Kubernetes?

You can run databases in K8s. But that means you handle backups, failover, storage scaling, and the 2 AM alerts. With AWS managed services, AWS handles all of that. For most teams, that trade-off is worth it.

Notice something? Every lesson follows the same pattern:

Tutorial approach → works in dev → breaks in production → costs money

Production approach → requires more setup → saves money → actually works at scale

This course exists to close that gap. Not with theory — with working Terraform configurations, Kubernetes manifests, and live demos you can follow along with.

All the code referenced in this newsletter is publicly available:

GitHub Repo: devops-real-world-project-implementation-on-aws

Star the repo if you find it useful!

Which of these five lessons did you learn the hard way? Hit reply and share your story — I read every response.

Kalyan Reddy Daida | StackSimplify