40 Monitoring and Observability Interview Questions (2026)

Observability is how you understand what's happening in production. When something breaks at 3 AM, your observability stack determines whether you fix it in minutes or spend hours guessing.

Interviewers test observability knowledge because it separates engineers who've operated real systems from those who've only built them. You can write perfect code, but if you can't debug it in production, you're not ready for senior roles.

This guide covers the three pillars of observability, the tools you'll be asked about, and the interview questions that test whether you actually understand monitoring or just know the buzzwords.

Table of Contents

1. Monitoring vs Observability Questions
2. Three Pillars of Observability Questions
3. Metrics and Prometheus Questions
4. PromQL Questions
5. RED and USE Method Questions
6. Grafana Questions
7. Logging Questions
8. ELK Stack Questions
9. Distributed Tracing Questions
10. OpenTelemetry Questions
11. Alerting Questions
12. SLI, SLO, and SLA Questions
13. Incident Response Questions
14. Observability Scenario Questions

Monitoring vs Observability Questions

These terms are often used interchangeably, but they represent different approaches to understanding system health.

What is monitoring and what questions does it answer?

Monitoring is the practice of collecting and evaluating signals against known health, capacity, and reliability questions. It can be reactive through incident alerts or proactive through trend analysis, forecasting, and SLO burn-rate alerts.

Typical monitoring questions include: Is CPU above 80%? Is the service responding? Are there more than 10 errors per minute? Monitoring tells you that something is wrong, but often lacks the context to explain why.

What is observability and how does it differ from monitoring?

Observability is the ability to infer a system's internal behavior from the signals it emits. Good instrumentation and queryable context make novel investigations possible, but no telemetry stack can answer an arbitrary question if the relevant data was never captured.

With an observable system, you can investigate: Which cohort is slow? What changed between yesterday and today? Where does a failure propagate? Observability supports those investigations; it does not automatically provide a single root cause.

The key difference comes down to exploration versus alerting. A well-monitored system has good dashboards for known metrics. An observable system has sufficient instrumentation to debug novel problems through exploration and correlation.

Monitoring: "Is the system healthy?" (yes/no)
Observability: "Why did user X's request fail at 2:34 PM?" (investigation)

What makes a system observable?

A system is observable when emitted signals and context let operators explain relevant internal behavior. Metrics, logs, and traces are common inputs, while profiles, events, topology, deployments, feature flags, and business context can be equally important.

Observable systems correlate signals with stable resource and request context and retain enough dimensions to investigate useful cohorts. High-cardinality attributes are valuable in event and trace stores but can be prohibitively expensive as metric labels; avoid secrets and unnecessary personal data in every signal.

Three Pillars of Observability Questions

Metrics, logs, and traces are a useful teaching model for complementary telemetry signals.

What are the three pillars of observability?

The traditional three pillars are metrics, logs, and traces. They often work best when correlated, but collecting all three does not guarantee useful observability and not every system needs the same mix. OpenTelemetry also defines baggage for propagating context, while profiles are an emerging signal.

How do the three pillars work together in practice?

In a typical debugging scenario, you move between pillars as you narrow down the problem. Each pillar provides context that guides you to the next level of detail.

A typical debugging flow demonstrates this interplay. First, a metrics alert fires showing the error rate spiked to 5%. Then you investigate logs, which reveal errors showing "database connection timeout." Finally, traces pinpoint that requests to /api/orders are slow specifically at the inventory service database call.

Metrics can be efficient at bounded cardinality but lose per-event detail. Logs can be rich but expensive to index and search. Traces describe only instrumented operations and may be sampled. Correlation improves investigation, but data quality, retention, cost, and coverage still set hard limits.

Metrics and Prometheus Questions

Metrics are numerical measurements collected over time. They are efficient for aggregation when label cardinality is controlled; unbounded labels such as user IDs or request IDs can make them very expensive.

What are the different Prometheus metric types?

Prometheus supports four metric types, each designed for different measurement scenarios. Understanding when to use each type is essential for interviews and for building effective monitoring.

When would you use a histogram vs a summary?

This is a common interview question that tests whether you understand the tradeoffs. The key difference is where percentile calculation happens and whether results can be aggregated.

Classic histograms count observations in configured cumulative buckets; native histograms use a dynamic bucket schema and are stable from Prometheus 3.8, though ingestion must still be enabled in Prometheus 3.x. PromQL calculates quantiles from histogram data, and compatible histograms can be aggregated across instances. Summaries calculate configured quantiles in the client, and those quantile series cannot be meaningfully aggregated.

Prefer native histograms when the client and pipeline support them and their resolution fits the accuracy requirement. Otherwise, design classic buckets around the SLO or use a summary only when client-side quantile error and a fixed window are acceptable and cross-instance aggregation is unnecessary.

How does Prometheus collect metrics?

Prometheus uses a pull-based model where it scrapes HTTP endpoints exposed by applications. Each application exposes a /metrics endpoint that returns current metric values in Prometheus format. The Prometheus server periodically scrapes these endpoints and stores the data in its time-series database.

This pull model makes failed scrapes visible and keeps target identity at the scraper. The Pushgateway is not a general replacement for scraping short-lived processes: Prometheus recommends it mainly for the outcome of service-level batch jobs because pushed series otherwise outlive the process and become stale unless explicitly deleted.

PromQL Questions

Prometheus Query Language (PromQL) is used by Prometheus and many other systems. Knowing common query patterns is essential for interviews.

What are the essential PromQL queries you should know?

PromQL has specific patterns for each metric type. The most important distinction is between instant vectors (single value per series) and range vectors (values over time).

Use rate() or increase() when a counter's change over time is the question; these functions account for resets. The raw cumulative value can still be useful for debugging or since-start totals, but it is usually unsuitable for alert thresholds.

# Rate of requests per second (for counters)
rate(http_requests_total[5m])
 
# Increase in requests over 1 hour
increase(http_requests_total[1h])

For gauges, the raw value is meaningful since gauges can go up or down.

# Current memory available
node_memory_available_bytes
 
# Average over 5 minutes
avg_over_time(node_memory_available_bytes[5m])

For histograms, use histogram_quantile() to calculate percentiles from bucket data.

# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

How do you filter and aggregate metrics in PromQL?

PromQL uses label matching for filtering and aggregation operators for grouping results. Labels are key-value pairs attached to each metric that enable slicing and dicing data.

Filtering uses curly braces with label matchers. The =~ operator enables regex matching.

# Filter by exact label value
http_requests_total{status="500"}
 
# Filter by regex (5xx errors)
http_requests_total{status=~"5.."}
 
# Exclude values
http_requests_total{method!="OPTIONS"}

Aggregation operators combine multiple series. The by clause specifies which labels to preserve.

# Sum all requests
sum(rate(http_requests_total[5m]))
 
# Sum by service
sum(rate(http_requests_total[5m])) by (service)
 
# Average by instance
avg(node_cpu_seconds_total) by (instance)

RED and USE Method Questions

These frameworks provide systematic approaches to monitoring different types of systems.

What is the RED method for monitoring services?

The RED method defines three golden signals for monitoring request-driven services like APIs and microservices. It focuses on user-facing behavior rather than infrastructure metrics.

RED stands for Rate (requests per second), Errors (failed requests per second), and Duration (latency distribution). These three metrics capture the essential user experience: how much traffic, how many failures, and how fast.

# Rate - requests per second
sum(rate(http_requests_total[5m]))
 
# Errors - error rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
 
# Duration - 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

RED is a service-centric starting point. Apply it to each meaningful request boundary, define errors from the user's perspective, and preserve latency distributions; not every change in traffic or duration implies user harm.

What is the USE method for monitoring infrastructure?

The USE method defines three metrics for monitoring infrastructure resources like CPU, memory, disk, and network. It focuses on resource health and capacity planning.

USE stands for Utilization (percentage of resource capacity used), Saturation (queue depth when resource is fully utilized), and Errors (error events for the resource). These three metrics reveal whether resources are the bottleneck.

When would you use RED vs USE?

Use RED for request-driven services such as APIs. Healthy RED metrics are useful evidence, but they can miss correctness, freshness, asynchronous workflows, and client-side failures.

Use USE for finite resources such as CPU, memory, disks, network links, pools, and queues. Healthy USE metrics reduce the likelihood of a resource bottleneck but do not prove end-to-end capacity.

In practice, you need both. A service might have good RED metrics while running on saturated infrastructure. Eventually the infrastructure problems will manifest as RED degradation, but USE gives you early warning.

Grafana Questions

Grafana is a widely used tool for visualizing and alerting on data from Prometheus and many other sources.

How do you design effective Grafana dashboards?

Dashboard design significantly impacts how quickly teams can diagnose issues. Good dashboards tell a story and guide investigation from high-level health to specific problems.

Start with the four golden signals at the top: latency, traffic, errors, and saturation. These give immediate visibility into service health. Group related metrics together logically—don't scatter CPU metrics across multiple rows.

Use consistent colors and naming conventions, accessible palettes, explicit units, and meaningful legends. Reserve red for states that need attention rather than ordinary series. Variables make dashboards reusable, and annotations correlate deployments or incidents with signal changes.

What are Grafana variables and why are they useful?

Grafana variables create dropdown selectors that parameterize dashboard queries. Instead of creating separate dashboards for each service or environment, one dashboard serves all by selecting the appropriate variable values.

Common variable types include query variables (populated from Prometheus labels), custom variables (static lists), and interval variables (for time-based aggregations). Variables are referenced in queries using $variable_name syntax.

# Query using variables
rate(http_requests_total{service="$service", environment="$environment"}[5m])

How does Grafana alerting work?

Grafana-managed alert rules evaluate queries independently of dashboard panels, although rules can link back to dashboards and panels. Evaluation groups control cadence and pending periods help avoid transient noise.

A basic alert checks if a query result crosses a threshold for a specified duration. For example, alert if error rate exceeds 1% for 5 minutes. The duration prevents alerting on brief spikes.

Contact points define destinations such as email, Slack, PagerDuty, or webhooks. Notification policies route and group alert instances by labels, while silences, mute timings, and inhibition control delivery. Templates can add values, runbooks, and dashboard links.

Logging Questions

Logs record discrete events and provide the detail that metrics lack, but they're expensive to store and query at scale.

What is the difference between structured and unstructured logs?

Structured logs use a consistent format (typically JSON) with defined fields, making them queryable and aggregatable. Unstructured logs are free-form text that requires parsing to extract information.

Unstructured logs are human-readable but machine-unfriendly:

2026-01-07 10:15:32 ERROR Failed to process order #12345

Structured logs are machine-parseable and queryable:

{
  "timestamp": "2026-01-07T10:15:32Z",
  "level": "error",
  "message": "Failed to process order",
  "order_id": "12345",
  "customer_segment": "business",
  "error": "payment_declined"
}

With structured logs, you can query by fields such as order_id or aggregate by error type. Avoid logging credentials, tokens, payment data, or unnecessary personal information; structured sensitive data is easier to search and leak.

What are log levels and when should you use each?

Log levels categorize messages by severity and help filter noise. Using levels consistently across services enables effective log analysis and alerting.

The distinction between WARN and ERROR is a common interview question. WARN means something unusual happened, but the system handled it—a retry succeeded, a fallback was used. ERROR means something actually failed and likely needs investigation or action.

What are correlation IDs and why are they important?

A correlation ID is an application-level identifier for related work; a trace ID identifies an OpenTelemetry/W3C trace. They can serve a similar diagnostic purpose but are not automatically interchangeable. Propagate trusted context across supported boundaries and record the trace ID and span ID in logs when available.

Use W3C Trace Context (traceparent and tracestate) for distributed tracing rather than inventing a tracing header. A separate business correlation ID can be useful across asynchronous or long-running workflows. Validate externally supplied identifiers and avoid using baggage for secrets or high-volume personal data.

{"correlation_id": "abc-123", "service": "api", "message": "Received order request"}
{"correlation_id": "abc-123", "service": "inventory", "message": "Checking stock"}
{"correlation_id": "abc-123", "service": "payment", "message": "Processing payment"}

ELK Stack Questions

The ELK stack (Elasticsearch, Logstash, Kibana) is a common logging solution, though many teams now use alternatives or cloud services.

How does the ELK stack work?

The historical ELK acronym refers to Elasticsearch, Logstash, and Kibana. The broader Elastic Stack also includes collection agents and integrations, and Logstash is optional when another collector sends data directly.

flowchart LR
    App["Application"] --> LS["Logstash<br/>(collect)"]
    LS --> ES["Elasticsearch<br/>(store/index)"]
    ES --> K["Kibana<br/>(visualize)"]

Elasticsearch is a distributed search and analytics engine. It stores logs in indices, enables full-text search and aggregations, and scales horizontally with sharding.

Logstash is a data processing pipeline. It collects logs from multiple sources, parses and transforms them (extracting fields, enriching data), and sends them to Elasticsearch.

Kibana is the visualization layer. It provides dashboards, log exploration interfaces, and alerting capabilities on top of Elasticsearch data.

What are common alternatives to the ELK stack?

Many teams replace Logstash with lighter alternatives like Filebeat or Fluent Bit. These agents are more resource-efficient for simple log collection and forwarding.

Managed logging services reduce some infrastructure operations but still require retention, access control, cost, and ingestion governance. Loki primarily indexes labels and stores compressed log chunks rather than building a full-text index of every line.

The choice depends on scale, budget, and operational capacity. Self-hosted ELK requires significant expertise to operate reliably at scale.

What log aggregation patterns are commonly used?

Centralized logging means all services send logs to a single system. This provides one place to search and enables correlation across services. The downside is network dependency and potential bottleneck.

The sidecar pattern runs a collector alongside a workload and is useful for special file formats or isolation, but adds one collector per pod. Kubernetes commonly uses a node-level agent or DaemonSet to collect container stdout/stderr instead.

The DaemonSet pattern runs a collector on each node and reads container logs exposed by the node runtime. It is usually more resource-efficient than sidecars, but needs careful permissions, buffering, multiline handling, and metadata enrichment.

Distributed Tracing Questions

Tracing shows how instrumented operations relate across distributed systems and is especially useful for latency and dependency investigations.

Why is distributed tracing important in microservices?

In a monolith, a stack trace shows where things went wrong. In microservices, a single request might touch dozens of services, and stack traces only show one service at a time. Tracing solves this by tracking requests across service boundaries.

Consider a request that crosses an API gateway and several services. Correlated spans can show recorded timing relationships across instrumented boundaries. Missing spans, asynchronous work, clock issues, queuing outside instrumentation, and sampling can still leave gaps, so traces are evidence rather than a perfect recording.

What are traces, spans, and context propagation?

A trace represents the complete journey of a request through a distributed system. It contains multiple spans representing individual operations.

A span represents a single operation within a trace—typically one service call. Each span includes start time, duration, service name, operation name, tags (metadata), and logs (events within the span).

Context propagation is passing trace and span IDs between services so spans can be correlated into a complete trace. Without propagation, you get isolated spans instead of connected traces.

flowchart TB
    T["Trace: order-request-abc123"]
    T --> AG["api-gateway<br/>(10ms)"]
    AG --> Auth["auth-service<br/>(5ms)"]
    AG --> Order["order-service<br/>(200ms)"]
    Order --> Inv["inventory-check<br/>(50ms)"]
    Order --> Pay["payment-process<br/>(140ms)<br/>← bottleneck"]

What are the different tracing sampling strategies?

Tracing every request is expensive at scale. Sampling reduces volume while maintaining visibility into system behavior. The strategy determines which requests get traced.

Tail-based sampling can retain recorded errors and slow traces after their outcome is known, while sampling ordinary traffic more aggressively. It requires buffering complete or near-complete traces, adds cost and latency, and cannot recover spans dropped earlier by head sampling or failed export.

OpenTelemetry Questions

OpenTelemetry (OTel) is a vendor-neutral CNCF project for telemetry APIs, SDKs, semantic conventions, protocols, and collection.

What is OpenTelemetry and why should you use it?

OpenTelemetry is a vendor-neutral observability framework that provides APIs, SDKs, and tools for generating and collecting telemetry data. It's the result of merging two earlier projects: OpenTracing and OpenCensus.

OpenTelemetry reduces vendor coupling through common APIs, semantic conventions, and OTLP. Backend changes may still require exporter, Collector, sampling, attribute, query, dashboard, and retention changes, especially when an application uses vendor-specific features.

OpenTelemetry provides auto-instrumentation for many languages and libraries, but signal maturity and coverage differ. Verify that important custom operations, messaging boundaries, errors, and resource attributes are captured rather than assuming an agent makes the system observable.

How does the OpenTelemetry Collector work?

The OpenTelemetry Collector is a vendor-agnostic proxy that receives, processes, and exports telemetry data. It decouples instrumentation from backends and provides a central point for processing.

flowchart LR
    App["App<br/>(OTel SDK)"] --> Collector["OTel<br/>Collector"]
    Collector --> Backend["Jaeger/Zipkin/<br/>Datadog"]

Collector pipelines use receivers, optional processors, and exporters. Connectors can link pipelines, while extensions add capabilities such as health checks. Reconfiguring the Collector can redirect compatible telemetry, but backend-specific schemas and dashboards may still need migration.

What is Jaeger and how is it used?

Jaeger is an open-source distributed tracing backend. Jaeger v2 is built on the OpenTelemetry Collector framework and can run collector, query, ingester, or all-in-one roles. Production storage and deployment choices depend on volume, durability, and the supported storage APIs.

Jaeger provides trace search and visualization, service dependency views, and analysis features that help investigate latency and recorded errors. These features narrow hypotheses; they do not automatically prove a root cause.

Jaeger can receive OTLP directly because Jaeger v2 is an OpenTelemetry Collector distribution. A separate Collector is optional and useful when teams also process metrics and logs, enrich telemetry, or centralize sampling before forwarding to Jaeger.

Alerting Questions

Observability data is useless if no one sees it when things break. Alerting bridges the gap between data and action.

What are the key principles of good alert design?

Well-designed alerts reduce noise while ensuring real issues get attention. Every notification needs an owner and purpose, and every page must call for timely human action—otherwise it belongs in a ticket, dashboard, or report.

Page primarily on user-visible symptoms or fast error-budget burn. Cause and capacity alerts can still be valuable when they predict imminent impact and have an immediate operator action. A fixed error-rate threshold also needs traffic volume, SLO semantics, and an evaluation window to be meaningful.

Route by urgency and required response. Page only when timely human action is necessary; create tickets or dashboard annotations for work that can wait. Severity names alone are less useful than explicit ownership, impact, and response expectations.

Include runbook links in alert descriptions. When someone gets paged at 3 AM, they shouldn't have to figure out what to do. The alert should link to step-by-step remediation instructions.

How do you prevent alert fatigue?

Alert fatigue occurs when teams receive so many alerts that they start ignoring them. It's dangerous because real issues get missed. Prevention requires discipline and regular review.

Review alerts regularly and delete ones nobody acts on. If an alert fires weekly and nobody investigates, it's training your team to ignore alerts. Set proper thresholds with hysteresis—alert at 90%, clear at 80%—to prevent flapping alerts.

Group related alerts so one incident doesn't generate 50 separate pages. Distinguish pages (wake someone up) from notifications (async awareness). Track alert metrics: acknowledge rate, false positive rate, time to resolution.

What is hysteresis in alerting and why is it important?

Hysteresis prevents alerts from flapping—repeatedly firing and clearing as a metric oscillates around a threshold. Without hysteresis, a CPU at 79-81% might trigger dozens of alerts per hour.

With hysteresis, you set different thresholds for firing and clearing. Alert when CPU exceeds 80%, clear when it drops below 70%. The metric must genuinely recover, not just dip briefly below the threshold.

This approach reduces noise while ensuring real issues still trigger alerts. The gap between thresholds should be large enough to indicate genuine state change.

SLI, SLO, and SLA Questions

These terms are frequently confused in interviews but represent distinct concepts in reliability engineering.

What is the difference between SLI, SLO, and SLA?

These three concepts form a hierarchy for defining and measuring service reliability. Understanding their relationships is essential for production operations.

SLIs measure actual performance—they're the metrics themselves. SLOs set internal targets for those metrics—they define "good enough." SLAs are promises to customers with consequences for missing them—they're usually looser than SLOs to provide buffer.

What is an error budget and how is it used?

For a ratio-based SLO, the error budget is the allowed bad-event fraction: 1 - SLO. A 99.9% target over a 30-day window permits 0.1% bad events. That equals about 43.2 minutes only for a time-based availability SLI under a simple full-outage model; request-based budgets are counted in requests, not minutes.

An agreed error-budget policy can balance change velocity and reliability. Remaining budget is not permission to ignore risky changes, and exhaustion should trigger the pre-agreed actions appropriate to the service rather than an automatic universal release freeze.

This approach replaces subjective debates ("Is this reliable enough?") with objective decisions based on measured reality. Teams that consistently exhaust their error budget know they need to invest in reliability.

How do you choose good SLIs?

Good SLIs measure what users actually experience, not internal system metrics. Focus on requests, not resources.

For request-driven services, common SLIs are the proportion of valid requests that succeed and the proportion that complete within a latency threshold. Throughput is usually workload context rather than a reliability SLI unless the user-facing promise explicitly concerns processing volume.

Avoid vanity metrics that look good but don't reflect user experience. 99.9% of requests succeeding is meaningful. 99.9% CPU availability is not—users don't experience CPU.

Incident Response Questions

Observability enables incident response, and interviewers often ask about on-call practices and postmortems.

What are on-call best practices?

Effective on-call requires clear processes, fair rotations, and good tooling. The goal is quick resolution with sustainable workload.

Rotation schedules should be fair and predictable. Weekly rotations are common, with clear handoffs documenting ongoing issues. Escalation paths define what happens when the primary doesn't respond—secondary on-call, then management.

Runbooks provide step-by-step remediation for common alerts. Good runbooks reduce mean time to resolution and enable less experienced engineers to handle incidents. They should be maintained as living documents updated after each incident.

What is a blameless postmortem?

A blameless postmortem analyzes an incident to understand what happened and prevent recurrence, without assigning individual blame. The focus is on systems and processes, not people.

The standard format covers: what happened (timeline), impact (users affected, duration), root cause (why it happened), contributing factors (what made it worse), and action items (what we'll change). Action items should be specific, assigned, and tracked.

Blameless culture is essential because blame discourages honesty. If people fear punishment, they'll hide mistakes rather than learning from them. The goal is to make the system more resilient, not to punish individuals.

How do you write effective incident timelines?

Incident timelines document what happened and when, providing the foundation for postmortem analysis. Good timelines are detailed, timestamped, and objective.

Include: when the incident was first detected, what alerts fired, when humans were engaged, what actions were taken, when mitigation began working, and when full recovery was confirmed. Note who did what, but avoid blame language.

Use UTC timestamps for clarity across time zones. Link to dashboards, logs, and chat transcripts that provide additional context. The timeline should enable someone who wasn't there to understand exactly what happened.

Observability Scenario Questions

Interviewers often present scenarios to test practical debugging skills.

Your API's p99 latency suddenly increased from 200ms to 2s. How would you investigate?

Start with metrics to understand the scope. Is it all endpoints or specific ones? All users or specific regions? Check recent deployments or config changes using dashboard annotations.

Next, examine representative traces to see where recorded time accumulates. A slow downstream span is a lead, not proof: compare it with healthy traces and check queueing, retries, sampling, and missing instrumentation.

Check resource metrics using the USE method—CPU saturation, memory pressure, disk I/O. Look for database slow queries in logs. Verify external dependency latency if you call third-party APIs.

The key is systematic investigation: narrow scope with metrics, identify bottleneck with traces, understand cause with logs and resource metrics.

How would you design alerting for a new microservice?

Start from user-facing SLIs and an SLO. Multi-window, multi-burn-rate alerts usually align paging better than one static threshold: a fast-burn page catches acute incidents, while a slower-burn notification catches sustained degradation. Adapt the design for low-traffic services where a single failure makes ratios volatile.

Add resource alerts only if they correlate with user impact. CPU alerts are often noisy—high CPU during legitimate load isn't a problem. Memory and disk alerts are more actionable.

Include runbook and dashboard links in every page. Configure escalation through the team's incident-management system. After collecting representative baseline data, review thresholds against SLOs, traffic patterns, and actual operator actions.

You're getting 1000 alerts per day and the team is ignoring them. How do you fix this?

This is an alert fatigue crisis. Start by auditing every alert that fired in the past week. Categorize them: actionable (someone investigated and fixed something), noisy (fired but no action taken), or duplicate (same incident, multiple alerts).

Delete alerts nobody acts on—they're training your team to ignore pages. Consolidate related alerts so one incident doesn't generate dozens of notifications. Separate pages (immediate action required) from notifications (async awareness).

Add hysteresis to prevent flapping. Track metrics going forward: acknowledge rate, time to resolution, false positive rate. Establish a regular review process—monthly alert pruning—to prevent accumulation.

Quick Reference

Common telemetry signals:

• Metrics: aggregated measurements and distributions
• Logs: discrete event records
• Traces: causally related operations across boundaries
• Profiles: code-level resource use; OpenTelemetry profile support is still maturing

RED Method (for services):

• Rate, Errors, Duration

USE Method (for resources):

• Utilization, Saturation, Errors

Metric Types:

• Counter (increases), Gauge (up/down), Histogram (distribution), Summary (percentiles)

SLI/SLO/SLA:

• SLI measures, SLO targets, SLA promises

Alert Best Practices:

• Page on actionable impact or imminent risk
• Actionable with runbooks
• Regular review and pruning

Related Articles

• Complete DevOps Engineer Interview Guide - Full DevOps interview preparation
• Docker Interview Guide - Container fundamentals
• Kubernetes Interview Guide - Container orchestration
• Linux Commands Interview Guide - Essential Linux skills
• CI/CD & GitHub Actions Interview Guide - Pipeline and deployment automation

Official References

• OpenTelemetry signals
• OpenTelemetry Collector components
• Prometheus metric types
• Prometheus histograms and summaries
• Prometheus: when to use the Pushgateway
• Grafana Alerting notifications
• Jaeger v2 architecture
• W3C Trace Context
• Google SRE Workbook: alerting on SLOs

Frequently Asked Questions

What is the difference between monitoring and observability?

Monitoring collects and evaluates signals for known health and reliability questions. Observability is a property of a system: its telemetry and context let operators infer internal behavior and investigate both expected and novel failures. The practices overlap; observability does not guarantee arbitrary answers without suitable instrumentation.

What are the three pillars of observability?

Metrics, logs, and traces are commonly called the three pillars, but they are complementary signals rather than a completeness checklist. Profiles, events, topology, deployment data, and business context may also be needed. OpenTelemetry currently supports traces, metrics, logs, and baggage, while profiles are still maturing.

What is the difference between SLI, SLO, and SLA?

An SLI is a quantitative measurement of service behavior, such as the proportion of valid requests completed successfully. An SLO is a target for that SLI over a defined window. An SLA is an agreement with users or customers and may specify remedies or other consequences when commitments are missed.

What is the RED method in monitoring?

RED stands for Rate, Errors, and Duration. It is a service-oriented starting point for request-driven workloads; define what counts as a request and an error, and observe a latency distribution rather than only an average. USE—Utilization, Saturation, Errors—complements it for resources.

How do you prevent alert fatigue?

Alert on symptoms not causes, set appropriate thresholds with hysteresis, group related alerts, ensure every alert is actionable, regularly review and prune alerts, and use severity levels appropriately. If an alert doesn't require human action, it shouldn't page anyone.

What is distributed tracing and why is it important?

Distributed tracing correlates spans across service and messaging boundaries to show a request's path, timing, and recorded errors. OpenTelemetry provides instrumentation, context propagation, and collection; Jaeger is one backend for receiving, storing, querying, and visualizing traces. Sampling and missing instrumentation limit what any trace can show.