A Beginner's Guide to Monitoring and Observability

Monitoring answers questions you thought of in advance. Observability is for the ones you did not.

A team had sixty dashboard panels across four screens. Every host on a graph. Every service wired up. They found out about their worst outage of the year from a customer email, forty minutes in. Nothing on any dashboard was red. The broken thing was a third party payment callback nobody had thought to graph, and the symptom was checkouts quietly not finishing.

That gap is not about tooling. It is about the difference between the two words in the title. This post covers what each one means, the three signals and when to reach for each, how to alert so people act instead of ignoring you, the cardinality mistake that takes down time series databases, and what to instrument first. Three failures follow, and each one needs a different signal to catch it.

What the two words mean

Monitoring watches for conditions you defined ahead of time. It tells you when something you predicted goes wrong.

Observability is the other half. It means you can ask new questions of a system later, and chase a failure nobody planned for, without shipping code first.

You need both. Monitoring catches known failures cheaply. Observability is what you want at three in the morning when the failure is strange.

Why the difference is practical

The argument over these two words gets dull fast. Here is the version that changes what you do.

A dashboard is a set of answers to questions someone already asked. That is genuinely useful. Most incidents are repeats, and a good dashboard solves them in seconds.

But a dashboard cannot cover the failure nobody imagined. The payment callback was exactly that shape.

Observability asks something else. Can the data you already collect answer a question you have not asked yet? So the real test is not which tools you run. It is whether you can chase a new problem without deploying anything. If your answer is always "let me add a metric and redeploy," you have monitoring. Not observability. And the difference costs you hours during the incidents that hurt most.

The three signals

Most guides list metrics, logs, and traces and stop. What you need is knowing which one answers which question. Reaching for the wrong signal is where incident time disappears.

Metrics are numbers rolled up over time. Requests per second. Memory used. They are cheap because a million requests become one number per interval. You alert on these. They cannot tell you about one individual request.

Logs are records of single events. They answer what happened to this specific user. They hold as much detail as you like, and they cost real money at volume, because every line gets stored.

Traces follow one request across every service it touched, timing each hop. In a system with more than a couple of services, this is the only signal that can tell you where the time went. They get sampled, because tracing every request is expensive.

The working pattern is simple. Metrics say something changed. Traces say where. Logs say exactly what.

Alert on symptoms, not causes

This one principle removes more pain than any tool you could buy.

A cause is something that might lead to a problem. High CPU. Low disk. A full queue. A pod restarting.

By contrast, a symptom is something a user would notice. Requests failing. Pages loading slowly. Checkouts not finishing.

Alert on causes and you get noise, because most causes cause nothing. CPU at ninety percent might just be a machine working hard. A restarting pod might be a normal deploy. Each one pages somebody, turns out to be nothing, and after enough of those people stop reading alerts at all. Now your alerting is worse than none, because it gives false comfort.

Now alert on symptoms instead. You get far fewer alerts, and every one matters, because a user is affected by definition.

That last row is the one people forget. Alerting on failure catches things that break loudly. Alerting on the absence of success catches things that stop running entirely, which produce no failures at all. A scheduled job that quietly stops firing generates zero errors forever.

The four golden signals

If you do not know what to instrument, start here. This works for almost any service and takes an afternoon.

Latency. How long requests take. Use percentiles, not averages. And split successful from failed requests, because a fast error is not a fast response.

Traffic. How much demand is arriving.

Errors. The rate of failing requests. Include the ones that return a success code while doing the wrong thing.

Saturation. How full the tight resource is. Memory, connections, a thread pool.

Two shorthands make this practical. RED suits request driven services: Rate, Errors, Duration. That is the golden signals minus saturation, and it is the right start for an API. USE suits resources. It stands for how full, how backed up, and how many errors. Good for queues, disks, and pools.

groups:
  - name: symptom-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (service) (rate(http_requests_total[5m])) > 0.02
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.service }} is failing over 2 percent of requests"
          runbook_url: "https://runbooks.internal/high-error-rate"

      - alert: LatencyRegression
        expr: |
          histogram_quantile(0.99,
            sum by (le, service) (rate(http_request_duration_seconds_bucket[10m]))) > 2
        for: 15m
        labels:
          severity: page

      - alert: NoRecentSuccess
        expr: |
          time() - max by (job) (nightly_job_last_success_timestamp_seconds) > 90000
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} has not succeeded in over 25 hours"

Three details there do the real work.

The for clause makes a condition persist before it fires. That kills the single blip that would otherwise wake someone. Leaving it out is the most common reason alerting feels noisy.

Notice the error rate is a ratio, not a count. So it means the same thing during a traffic spike as on a quiet night.

And the third rule watches for missing success rather than present failure. That is how you catch a job that stopped running, not one that ran and failed. Note it has no for clause, on purpose. The condition already waits twenty five hours.

The cardinality trap

This is the mistake that turns a monitoring project into an incident. Worth knowing before you instrument anything.

A time series database stores one series for every unique set of label values. Three labels with ten, twenty, and five options each already make a thousand series.

Trouble starts when a label has no fixed range. Customer names. Request IDs. Session IDs. URLs with numbers baked into them. All of those grow as your traffic grows. Attach one to a metric and your stored series climb forever, until memory runs out. And your monitoring dies exactly when you most want it.

So the rule is a hard line, not a preference. A metric label must come from a short list you could write down. Which service. Which status class. Which region. Which method. Anything that varies per request goes in a log line or a trace tag. Both are built for that.

That route row deserves care. A label holding /orders/12345 has no limit, because the number changes. A label holding /orders/{id} does, because it is the route template. One line of instrumentation code. The difference between a healthy database and an outage.

Build the Prometheus foundation properly

All of this gets concrete the moment you scrape a target, write PromQL, and watch an alert fire. The Prometheus Certified Associate (PCA) prep course on KodeKloud covers observability fundamentals, Prometheus architecture, exporters, PromQL, alerting, and Kubernetes monitoring, with a lab after every topic.

SLIs, SLOs, and what they are for

These acronyms sound like paperwork. They solve a real engineering problem.

An SLI is a number measuring something users care about. Say, the share of requests that succeed within a second. An SLO is the target you set for it. Say, 99.5 percent over thirty days. The gap between your target and 100 percent is your error budget. That is the failure you have agreed to live with.

Their job is deciding what to work on next. Inside your budget? Things are good enough, so ship features. Out of budget? Fixing comes first. That turns an argument about whether things feel stable into a number everyone already agreed to.

Two practical notes. Pick a target you can justify rather than chasing nines, because each extra nine costs far more and users often cannot tell. And measure where the user is, at your edge, not inside your service. A backend reporting perfect health while a load balancer returns errors is common.

Three failures, three signals

Same toolkit. Three failures, each needing a different signal.

The deploy that looked fine

A release goes out. Error rates stay flat. Latency looks normal. Then support tickets arrive about a feature showing stale data.

Why metrics miss it. Nothing is failing in a way HTTP status codes can express. The service returns 200 with the wrong content. That is the shape of the story at the top of this post.

What finds it. A business metric alongside the technical ones. Something a product person would recognise, like cache hit ratio for that feature, or the rate of a write that should follow every read. Technical metrics tell you the machinery works. Only a business metric tells you it is doing the right thing.

What to add. For any feature where a wrong answer is possible without an error, emit one counter for the outcome, not the request. Orders created. Emails sent. Records synced. One line of code, and the only thing that would have caught this.

The latency spike that comes and goes

Every few hours, 99th percentile latency triples for ten minutes, then recovers. Nothing lines up with deploys. CPU looks fine throughout.

Why you need traces. This is where their absence costs the most. Metrics tell you latency rose. In a system with eight services, they cannot tell you which hop ate it. So without tracing you are checking services one at a time, chasing a symptom that has already gone.

What a trace shows. One slow request, broken down by service, with time attributed to each. The usual culprits found this way: a downstream dependency you forgot was in the path, a connection pool exhausting under a periodic batch job, or a cache expiring all at once and sending everything to the database.

About sampling. Tracing everything is expensive, so you sample. But make sure your sampling keeps the slow requests. Tail based sampling decides after a request finishes, so it can keep every trace above a latency threshold. That is exactly the population you want, and the opposite of what random sampling gives you.

The monitoring system that fell over

Someone added a label to track usage per customer. Two days later Prometheus is eating all available memory and queries time out. Your monitoring is down during the period you most need it.

Why it happened. This is the cardinality trap, and it is a platform failure rather than an application one. A customer ID on a metric label made one series per customer, per metric, per other label. The maths did the rest.

What to do right now. Find the offending metric by series count, not by guessing. Drop the label as metrics arrive, using a relabelling rule, while you fix the instrumentation. Then set limits so next time it gets rejected rather than fatal.

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'api_requests_total'
    action: labeldrop
    replacement: 'customer_id'

scrape_configs:
  - job_name: 'api'
    sample_limit: 10000
    label_limit: 15
    label_value_length_limit: 128

The relabelling rule stops the bleeding without waiting for an application deploy. The three limits underneath are the real fix. With sample_limit set, a scrape fails loudly instead of quietly swallowing a million new series. A failed scrape beats a database that will not start.

The wider lesson. No single team can see the whole picture here. So the limits belong in the platform, on by default. A rejected scrape beats a shared outage.

Practise on a live stack

This is a hands on skill. The ideas land much faster with a real Prometheus to query and a real Grafana to build in. The Prometheus and Grafana playground on KodeKloud gives you both running and pre configured, so you can write PromQL and build panels with no setup at all.

What to instrument first

Starting from nothing? This order gets you the most value soonest.

Begin with the golden signals on your most important service. For a request driven service that means RED: rate, errors, duration. Three metrics, and they will already catch most real incidents.

Add one business metric next. A counter for the outcome that service exists to produce. This is the step that separates useful monitoring from technically complete monitoring, and it is the one most teams skip.

Then add structured logging. Key value pairs rather than sentences, with a request ID on every line. Structured logs are searchable and can be joined up. Plain text logs are neither, at any scale.

Add tracing once a request path crosses more than two or three services. That is where metrics stop being able to tell you where a problem lives.

Finally, write alerts for symptoms only, and give every one a runbook link. That link matters more than it sounds. An alert at three in the morning with a link to the procedure is a very different experience from the same alert alone.

Try this on your own systems

Give it an hour. Every step works on a system you already run.

Start by asking your dashboards a question they were not built for. Something plausible, like which customers were affected during last week's slow period. Try to answer it with what you already collect. Whether you can is your observability, and this tells you more than any tool comparison.

Then count last month's alerts. For each one, write down what the person did about it. Alerts that led to nothing are candidates for deletion. The count usually surprises people, and deleting a noisy alert is a real improvement.

Next, look at your highest cardinality metric. Most systems can report series counts by metric name, and the top of that list is almost always a label someone added without thinking about multiplication. Check whether any route label holds IDs instead of templates.

Finally, pick your most important service. Do you have a metric describing what it is for, rather than what it does? Not requests served, but orders placed, messages delivered, files processed. If not, add one this week. That is the metric that would have caught the outage at the top of this post.

What you should be able to answer now

When something breaks, which signal do you reach for first? Metrics to know something changed. Traces to know where. Logs to know exactly what. Teams that grab logs first spend incidents searching text when they could read a number.

How many of your alerts led to action last month? A poor ratio means you have alert fatigue, not coverage. The fix is deleting alerts, not adding more.

Which of your metrics could take down your database? Any label holding a user ID, a request ID, or a URL with IDs in it. Worth checking today, not during the incident.

For your most important service, what would tell you it is doing the wrong thing correctly? That is the business metric question, and it is what the opening story turns on.

The team with sixty panels did a lot of work. What they lacked was one counter for completed checkouts. That would have alerted in ninety seconds instead of forty minutes, and no amount of infrastructure graphing could stand in for it. Dashboards are not useless. They just answer the questions someone thought to ask. The outage that hurts is the one nobody thought of.

Ready to Build Real Observability Skills?

This rewards practice, because the ideas are simple and the judgment comes from debugging real systems. The Prometheus Certified Associate (PCA) prep course on KodeKloud covers fundamentals, PromQL, exporters, and alerting in depth. The Grafana Loki course covers the logging half, including Promtail and querying. And the Prometheus and Grafana playground gives you both running instantly. Start with the signal you are weakest at.

FAQs

Q1: What is the difference between monitoring and observability?

Monitoring watches for conditions you defined in advance, so it tells you when something you predicted goes wrong. A dashboard is a set of answers to questions someone already asked. Observability is the property that lets you ask new questions after the fact, so you can chase a failure nobody planned for without shipping code first. You need both. Monitoring catches known failures cheaply. Observability is what you want when the failure is strange. The real test is not which tools you run. It is whether you can chase a new problem with data you already collect. If your answer is always "let me add a metric and redeploy," you have monitoring rather than observability. That difference costs you hours during the incidents that hurt most. The word itself comes from control theory, where it means working out a system's inner state from its outputs. That framing beats the marketing one.

Q2: What are metrics, logs, and traces, and when do I use each?

Metrics are numbers rolled up over time, like requests per second. They are cheap, because a million requests become one number per interval. That makes them what you alert on. They cannot tell you about one individual request. Logs are records of single events. They answer what happened in this specific case, hold any amount of detail, and cost real money at volume, because every line gets stored. Traces follow one request across every service it touched and time each hop. In a system with several services, that is the only way to see where time went. The working pattern is simple. Metrics say something changed. Traces say where. Logs say exactly what. The expensive mistake is making one do another's job. Alerting on log patterns is slower and pricier than alerting on a number. Putting per request detail in metric labels breaks your database.

Q3: Why do my alerts get ignored, and how do I fix it?

Almost certainly because they alert on causes rather than symptoms. A cause is something that might lead to a problem: high CPU, low disk, a restarting pod. Most causes cause nothing. High CPU is often just a machine working hard. A restarting pod is often a normal deploy. Each one pages somebody, turns out to be nothing, and after enough of those people stop reading. Now your alerting is worse than none, because it gives false comfort. Symptoms are things a user would notice: requests failing, pages loading slowly, checkouts not finishing. Alert on those and you get far fewer alerts, and every one matters. Two habits fix most of this. Before adding an alert, ask what the recipient should do, and if the answer is "look at it, probably nothing," make it a dashboard panel. And add a for clause so a condition persists before firing, since leaving it out is the top reason alerting feels noisy.

Q4: What should I instrument first?

The four golden signals on your most important service. For a request driven service that means RED: rate, errors, and duration. Three metrics, and they already catch most real incidents. Then add one business metric, a counter for the outcome that service exists to produce. Orders placed. Messages delivered. Technical metrics tell you the machinery works. Only a business metric tells you it is doing the right thing. This is the step most teams skip, and it is what catches a service returning success while producing rubbish. Next add structured logging, key value pairs rather than sentences, with a request ID on every line. Structured logs are searchable and can be joined up. Plain text is neither at scale. Add tracing once a request path crosses more than two or three services, because that is where metrics stop telling you where a problem lives. Then write alerts for symptoms only, each with a runbook link.

Q5: What is cardinality, and why does everyone warn about it?

A time series database stores one series per unique combination of label values. So three labels with ten, twenty, and five options make a thousand series. The danger is a label with no fixed range. A user ID. A request ID. A customer name. A URL with numbers baked in. Attach one of those to a metric label and your series count climbs with usage until memory runs out. Then your monitoring dies at exactly the moment you most want it. The rule is a hard line. Metric labels take values from a short list you could write down: service name, status class, region, method. Anything with no fixed range belongs in logs or trace tags, which are built for it. One distinction is worth learning. A label holding /orders/12345 has no limit, because the number changes. A label holding /orders/{id} does, because it is the route template. That is one line of code.

Q6: What are SLIs and SLOs, and do small teams need them?

An SLI is a number measuring something users care about, like the share of requests succeeding within a second. An SLO is the target you set for it, say 99.5 percent over thirty days. The gap between that and 100 percent is your error budget, which is the failure you agreed to live with. Their job is deciding what to work on. Inside your budget, things are good enough, so ship features. Outside it, fixing things comes first. That turns an argument about whether things feel stable into a number everyone already agreed to. Small teams benefit, with two cautions. Pick a target you can justify rather than chasing nines, because each extra nine costs far more and users often cannot tell. And measure at the edge where users are, not inside your service. The failure to avoid is setting a target nobody plans to act on, because that teaches everyone the numbers are theatre.