29 Apache Kafka Interview Questions (2026)

Apache Kafka is a common foundation for event-driven architectures, real-time data pipelines, and durable stream processing. This guide was reviewed against Apache Kafka 4.3.1, the current supported release on September 2, 2026. Kafka 4.x is KRaft-only, and the next-generation consumer group protocol introduced in 4.0 changes how modern rebalance answers should be framed.

This guide covers the Kafka concepts that interviewers actually ask about—from core fundamentals to production operational knowledge.

Table of Contents

1. Kafka Architecture Questions
2. Topic and Partition Questions
3. Producer Questions
4. Consumer Questions
5. Exactly-Once Semantics Questions
6. Kafka Streams Questions
7. ksqlDB Questions
8. Spring Kafka Questions
9. Replication and Reliability Questions
10. Performance Tuning Questions
11. Monitoring and Operations Questions

Kafka Architecture Questions

Understanding Kafka's architecture is the foundation for all advanced topics.

What are the main components of Kafka architecture?

Kafka's architecture consists of several interconnected components that work together to provide distributed, fault-tolerant event streaming. At its core, a Kafka cluster contains multiple brokers that store data and serve client requests. Topics organize messages into logical categories, while partitions enable horizontal scaling and parallel processing.

Each partition has a leader replica that handles client reads and writes, with follower replicas providing fault tolerance. Consumer groups coordinate multiple consumers so a partition is assigned to one member at a time. That assignment provides parallelism, not an automatic exactly-once processing guarantee; failures around processing and offset commits can cause replay or loss depending on the application design.

flowchart TB
    subgraph cluster["Kafka Cluster"]
        direction LR
        subgraph B1["Broker 1"]
            P0L["Topic A Part 0<br/>(Leader)"]
            P1R["Topic A Part 1<br/>(Replica)"]
        end
        subgraph B2["Broker 2"]
            P1L["Topic A Part 1<br/>(Leader)"]
            P0R1["Topic A Part 0<br/>(Replica)"]
        end
        subgraph B3["Broker 3"]
            P2L["Topic A Part 2<br/>(Leader)"]
            P0R2["Topic A Part 0<br/>(Replica)"]
        end
    end
 
    Producer["Producer"] --> cluster
    cluster --> CG["Consumer<br/>Group"]

Key components:

• Broker: A Kafka server that stores data and serves clients
• Topic: A category/feed name to which records are published
• Partition: An ordered, immutable sequence of records within a topic
• Replica: A copy of a partition for fault tolerance
• Leader: The replica that handles all reads and writes for a partition
• Consumer Group: A set of consumers that cooperate to consume a topic

What was ZooKeeper's role, and what is KRaft?

Apache ZooKeeper historically stored Kafka cluster metadata and participated in controller election and broker coordination. It required operators to deploy and secure a separate quorum, adding operational complexity.

KRaft integrates the metadata quorum into Kafka. Dedicated or combined controller nodes maintain a replicated metadata log using Kafka's Raft-based protocol. KRaft was declared production-ready in Kafka 3.3; ZooKeeper mode was deprecated in 3.5 and removed in Kafka 4.0, so every Kafka 4.x cluster runs in KRaft mode.

# Simplified KRaft combined broker/controller example
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093,2@localhost:9094,3@localhost:9095

Operational effects of KRaft:

• Simplified operations (one system instead of two)
• A more scalable metadata control plane without a separate ZooKeeper ensemble
• Faster controller failover
• Reduced operational complexity

Interview tip: In 2026, say that ZooKeeper is historical. Kafka 4.x is KRaft-only; a ZooKeeper-mode cluster must migrate before upgrading to 4.x.

Topic and Partition Questions

Topics and partitions are fundamental to understanding how Kafka scales and maintains ordering guarantees.

What is a Kafka topic and how does it differ from a traditional message queue?

A Kafka topic is a logical channel for publishing and subscribing to streams of records, functioning as the primary abstraction for organizing data. Unlike traditional message queues where messages are deleted after consumption, Kafka retains messages based on configurable retention policies, allowing multiple consumers to read the same data and enabling replay from any point in time.

This fundamental difference in data retention model makes Kafka suitable for use cases like event sourcing, audit logging, and stream processing where historical data access is valuable.

How do partitions affect message ordering?

Kafka guarantees message ordering only within a single partition, not across an entire topic. When a producer sends messages with the same key, Kafka hashes the key to determine the partition, ensuring all messages with that key go to the same partition and are therefore ordered. Messages with different keys may go to different partitions and can be processed in any order relative to each other.

This design enables parallel processing while maintaining ordering where it matters—typically by using entity identifiers as keys so that all events for a single entity are processed in order.

// Messages with same key go to same partition (ordered)
producer.send(new ProducerRecord<>("orders", "customer-123", order1));
producer.send(new ProducerRecord<>("orders", "customer-123", order2));
// order1 always processed before order2 for customer-123
 
// Messages with different keys may go to different partitions
producer.send(new ProducerRecord<>("orders", "customer-456", order3));
// order3 may be processed before order1 or order2

Partition assignment formula (default):

partition = hash(key) % numberOfPartitions

How many partitions should a topic have?

Choosing the right partition count requires balancing several factors. More partitions enable greater parallelism since each partition can be consumed by a separate consumer, but they also increase broker memory usage, extend recovery time after failures, and can add end-to-end latency due to more leader elections and replication overhead.

There is no portable "MB/s per partition" constant. Benchmark the actual record size, compression, replication factor, storage, network, producer batching, and consumer work. Start from required consumer parallelism and measured per-partition throughput, include growth and failure headroom, and remember that increasing a topic's partition count can change key-to-partition mapping.

Considerations:

• Parallelism: More partitions = more parallel consumers
• Throughput: Measure it with your workload and hardware
• Latency: More partitions = more end-to-end latency
• Memory: Each partition uses broker memory
• Recovery time: More partitions = longer recovery after broker failure

Producer Questions

Understanding producer configuration is critical for reliable message delivery.

What are the different acks settings and their trade-offs?

The acks setting controls what the producer requires before a send succeeds. With acks=0, the client does not wait for a broker response. With acks=1, the leader acknowledges after its local append, so an immediate leader failure before replication can lose the record. With acks=all (the modern producer default), the leader waits for every replica currently in the ISR; durability then also depends on replication factor, min.insync.replicas, and whether unclean leader election is allowed.

For most production use cases, acks=all combined with min.insync.replicas=2 provides the right balance of durability without sacrificing too much performance.

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
 
// acks=0: Fire and forget (fastest, least reliable)
props.put("acks", "0");
 
// acks=1: Leader acknowledgment (balanced)
props.put("acks", "1");
 
// acks=all: All in-sync replicas acknowledge (slowest, most reliable)
props.put("acks", "all");

What is an idempotent producer and why is it important?

An idempotent producer prevents duplicates caused by producer retries within the producer session. Without idempotence, a retry after an ambiguous timeout can append the same logical record twice. Kafka assigns a producer ID and sequence numbers so brokers can reject duplicate batches while preserving per-partition order. Idempotence is enabled by default when configurations do not conflict; it does not deduplicate an application that deliberately sends the same business event twice.

Enabling idempotence is straightforward and has minimal performance overhead, making it a best practice for all production deployments where message duplication would be problematic.

props.put("enable.idempotence", "true");
// Automatically sets:
// - acks=all
// - retries=Integer.MAX_VALUE
// - max.in.flight.requests.per.connection=5

How it works:

1. Producer gets a unique Producer ID (PID) on initialization
2. Each message gets a sequence number
3. Broker deduplicates based on PID + sequence number
4. Retried messages with same sequence are ignored

How does Kafka batching work and how do you tune it?

Kafka producers batch multiple messages together before sending them to brokers, improving throughput by reducing network round trips and enabling more efficient compression. The batch.size parameter sets the maximum batch size in bytes, while linger.ms controls how long the producer waits for additional messages before sending a partially-filled batch.

With linger.ms=0 (the default), messages are sent immediately without waiting for batching. Increasing this value allows more messages to accumulate, improving throughput at the cost of higher latency. Compression further improves efficiency by reducing network bandwidth and storage requirements at the cost of CPU overhead.

// Batch size in bytes
props.put("batch.size", 16384);  // 16 KB default
 
// Maximum wait time to fill batch
props.put("linger.ms", 5);  // Wait up to 5ms for more messages
 
// Compression
props.put("compression.type", "lz4");  // Options: none, gzip, snappy, lz4, zstd

Batching trade-offs:

• Larger batches: Better throughput, higher latency
• Smaller batches: Lower latency, more overhead
• Compression: Reduces network/storage but adds CPU overhead

How do you control which partition a message goes to?

Kafka provides three mechanisms for controlling partition assignment. The default behavior uses key-based partitioning where the hash of the message key determines the partition—this ensures all messages with the same key go to the same partition. For special cases, you can explicitly specify a partition number or implement a custom partitioner that applies business logic to route messages.

Custom partitioners are useful for scenarios like routing high-priority messages to dedicated partitions or implementing geographic routing based on message content.

// 1. Key-based (default): hash(key) % partitions
producer.send(new ProducerRecord<>("topic", "key", "value"));
 
// 2. Explicit partition
producer.send(new ProducerRecord<>("topic", 2, "key", "value"));
 
// 3. Custom partitioner
public class CustomPartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
 
        // Route VIP customers to dedicated partition
        if (key.toString().startsWith("vip-")) {
            return 0;  // VIP partition
        }
 
        // Default: hash-based distribution for others
        int regularPartitionCount = partitions.size() - 1;
        if (regularPartitionCount < 1) {
            throw new IllegalStateException("VIP routing requires at least 2 partitions");
        }
        return Math.floorMod(key.hashCode(), regularPartitionCount) + 1;
    }
}

Consumer Questions

Consumer configuration affects reliability, throughput, and exactly-once semantics.

How do consumer groups enable scalable consumption?

Consumer groups provide Kafka's mechanism for parallel message processing with automatic load balancing. When multiple consumers join the same group, Kafka distributes partitions among them so each partition is assigned to exactly one consumer. This enables horizontal scaling—add more consumers to process messages faster, up to the number of partitions.

Different consumer groups operate independently, each maintaining their own offset position and receiving all messages from subscribed topics. This enables the publish-subscribe pattern where multiple applications can independently consume the same data stream.

flowchart TB
    subgraph topic["Topic: orders (3 partitions)"]
        direction LR
        P0["Part 0"]
        P1["Part 1"]
        P2["Part 2"]
    end
 
    subgraph groupA["Consumer Group A"]
        direction LR
        C1A["Consumer 1<br/>Part 0"]
        C2A["Consumer 2<br/>Part 1, 2"]
    end
 
    subgraph groupB["Consumer Group B"]
        C1B["Consumer 1<br/>Part 0, 1, 2"]
    end
 
    P0 --> C1A
    P1 --> C2A
    P2 --> C2A
 
    P0 --> C1B
    P1 --> C1B
    P2 --> C1B

Key rules:

• Each partition assigned to exactly one consumer per group
• Multiple groups can consume same topic independently
• Max effective consumers = number of partitions

How do consumers track their position with offsets?

Consumers track their position in each partition with offsets. Kafka stores group commits in the internal __consumer_offsets topic so a reassigned or restarted consumer can resume. Auto-commit advances offsets periodically from the consumer's poll flow; the actual loss/duplication behavior still depends on whether processing completed before the committed position advanced.

Manual commit provides finer control. Committing only after successful processing normally gives at-least-once behavior: a crash after the side effect but before the commit replays the record. Exactly-once requires a transaction boundary that includes the relevant state change, or idempotency in the external system.

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "my-consumer-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
 
// Auto commit (default); semantics still depend on processing boundaries
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "5000");
 
// Manual commit for more control
props.put("enable.auto.commit", "false");
 
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("orders"));
 
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        processRecord(record);
    }
    // Synchronous commit (blocks until confirmed)
    consumer.commitSync();
 
    // Or async commit (non-blocking)
    consumer.commitAsync((offsets, exception) -> {
        if (exception != null) {
            log.error("Commit failed", exception);
        }
    });
}

What happens when a new consumer joins a group?

When membership or subscriptions change, Kafka redistributes partitions. The classic eager protocol can revoke all assignments and create a stop-the-world pause. Cooperative assignors reduce disruption by transferring only affected partitions incrementally.

Kafka 4.0 also made the next-generation consumer group protocol (KIP-848) generally available. It is fully incremental and removes the classic global synchronization barrier. Brokers support it automatically, while clients opt in with group.protocol=consumer; assignment and heartbeat/session settings then move to server-side configuration.

// Enable cooperative rebalancing
props.put("partition.assignment.strategy",
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
 
// Or opt into the Kafka 4.x next-generation consumer protocol
props.put("group.protocol", "consumer");

Rebalance strategies:

• Eager: Stop all, reassign all (default before 2.4)
• Cooperative (Incremental): Only reassign affected partitions
• Consumer protocol (Kafka 4.x opt-in): Fully incremental, server-side assignment

Exactly-Once Semantics Questions

Exactly-once claims must name the boundary. Kafka can atomically commit consumed offsets and records produced back to Kafka; it cannot automatically include an arbitrary database, email service, or HTTP API in that transaction.

How do you achieve exactly-once processing with Kafka?

Kafka's idempotent producer prevents duplicate appends caused by retries in a producer session. A transactional producer makes writes across Kafka partitions atomic. In a Kafka-to-Kafka read-process-write flow, adding consumed offsets to the same transaction and reading outputs with isolation.level=read_committed gives exactly-once processing semantics within Kafka.

External side effects remain outside that transaction. Use an idempotency key, an outbox/inbox pattern, or a datastore-specific transaction strategy when processing also changes another system.

Level 1: Idempotent producer (producer → Kafka)

props.put("enable.idempotence", "true");

Level 2: Transactional producer (across partitions)

props.put("transactional.id", "my-transactional-id");
 
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();
 
try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("topic1", "key", "value1"));
    producer.send(new ProducerRecord<>("topic2", "key", "value2"));
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
}

Level 3: Read-process-write (end-to-end)

// Consumer reads with isolation
props.put("isolation.level", "read_committed");
 
// Process and produce atomically
producer.beginTransaction();
for (ConsumerRecord<String, String> record : records) {
    ProducerRecord<String, String> output = process(record);
    producer.send(output);
}
// Commit consumer offsets as part of transaction
producer.sendOffsetsToTransaction(offsets, consumerGroupId);
producer.commitTransaction();

Kafka Streams Questions

Stream processing allows real-time transformations on Kafka data.

What is Kafka Streams and how does it differ from other stream processors?

Kafka Streams is a lightweight Java library for building stream processing applications, not a separate cluster infrastructure like Apache Flink or Spark Streaming. Applications built with Kafka Streams run as standard JVM applications—you simply start multiple instances to scale horizontally, and Kafka handles partition assignment and rebalancing automatically. This eliminates the operational overhead of managing a separate stream processing cluster.

The library provides exactly-once processing semantics, automatic state management with fault-tolerant local stores, and seamless integration with Kafka's partitioning model for parallel processing.

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "order-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
 
StreamsBuilder builder = new StreamsBuilder();
 
// Read from topic as stream
KStream<String, Order> orders = builder.stream("orders");
 
// Transform: filter, map, flatMap
KStream<String, Order> highValueOrders = orders
    .filter((key, order) -> order.getAmount() > 1000)
    .mapValues(order -> enrichOrder(order));
 
// Write to output topic
highValueOrders.to("high-value-orders");
 
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

Advantages:

• No separate cluster needed
• Exactly-once processing built-in
• Elastic scaling (just start more instances)
• Fault-tolerant with automatic recovery

What is the difference between KStream and KTable?

KStream and KTable represent two different ways of interpreting data in Kafka Streams. A KStream treats each record as an independent event in an append-only log—every record matters, even multiple records with the same key. A KTable interprets records as updates to a changelog, maintaining only the latest value for each key as current state.

This distinction is crucial for joins: joining a stream with a table enriches events with the latest lookup value, while joining two streams correlates events that occur within a time window.

// KStream: Append-only stream of events
// Each record is independent
KStream<String, PageView> pageViews = builder.stream("page-views");
// Records: (user1, view1), (user1, view2), (user2, view1)
 
// KTable: Changelog stream, latest value per key
// Represents current state
KTable<String, UserProfile> users = builder.table("users");
// State: {user1: profile1, user2: profile2}
 
// Join stream with table (enrichment)
KStream<String, EnrichedPageView> enriched = pageViews.join(
    users,
    (pageView, profile) -> new EnrichedPageView(pageView, profile)
);

How do you perform windowed aggregations in Kafka Streams?

Windowed aggregations group events by time intervals, essential for computing metrics like "orders per hour" or "page views in the last 5 minutes." Kafka Streams supports three window types: tumbling windows are fixed-size, non-overlapping intervals; hopping windows are fixed-size but overlap; and session windows group events by activity periods separated by gaps of inactivity.

Each window type serves different use cases—tumbling for hourly reports, hopping for moving averages, and session for user activity analysis.

KStream<String, Transaction> transactions = builder.stream("transactions");
 
// Tumbling window: fixed, non-overlapping
KTable<Windowed<String>, Long> hourlyCounts = transactions
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1)))
    .count();
 
// Hopping window: fixed size, overlapping
KTable<Windowed<String>, Long> slidingCounts = transactions
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeAndGrace(
        Duration.ofMinutes(5),
        Duration.ofMinutes(1)
    ).advanceBy(Duration.ofMinutes(1)))
    .count();
 
// Session window: activity-based, variable size
KTable<Windowed<String>, Long> sessionCounts = transactions
    .groupByKey()
    .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30)))
    .count();

ksqlDB Questions

ksqlDB provides SQL-based stream processing for teams that prefer declarative syntax.

When would you use ksqlDB instead of Kafka Streams?

ksqlDB is a streaming SQL engine that runs as a separate server, allowing you to write stream processing logic using familiar SQL syntax rather than Java code. This makes it ideal for rapid prototyping, simpler transformations, and teams with strong SQL expertise but less Java experience. ksqlDB also provides interactive querying capabilities for exploring streaming data.

However, for complex business logic, custom serialization, or when embedding stream processing directly in an application, Kafka Streams offers more flexibility and control.

-- Create a stream from a topic
CREATE STREAM orders (
    order_id VARCHAR KEY,
    customer_id VARCHAR,
    amount DECIMAL(10,2),
    order_time TIMESTAMP
) WITH (
    KAFKA_TOPIC='orders',
    VALUE_FORMAT='JSON'
);
 
-- Real-time aggregation
CREATE TABLE hourly_revenue AS
SELECT
    customer_id,
    WINDOWSTART as window_start,
    SUM(amount) as total_revenue,
    COUNT(*) as order_count
FROM orders
WINDOW TUMBLING (SIZE 1 HOUR)
GROUP BY customer_id
EMIT CHANGES;
 
-- Join streams
CREATE STREAM enriched_orders AS
SELECT
    o.order_id,
    o.amount,
    c.name as customer_name
FROM orders o
JOIN customers c ON o.customer_id = c.customer_id
EMIT CHANGES;

Use ksqlDB when:

• SQL is preferred over Java code
• Rapid prototyping needed
• Simpler transformations and aggregations
• Team has SQL expertise

Use Kafka Streams when:

• Complex business logic required
• Need full programmatic control
• Custom serialization/processing
• Embedding in existing application

Spring Kafka Questions

Spring Kafka simplifies Kafka integration in Spring Boot applications.

How do you configure Kafka in Spring Boot?

Spring Boot provides auto-configuration for Kafka through application properties, eliminating most boilerplate code. You specify bootstrap servers, serializers, and consumer group settings in YAML or properties files, and Spring creates the necessary producer and consumer factories automatically. For JSON serialization, Spring Kafka provides built-in serializers that handle object-to-JSON conversion with configurable trusted packages for deserialization security.

# application.yml
spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all
      retries: 3
    consumer:
      group-id: my-group
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      auto-offset-reset: earliest
      properties:
        spring.json.trusted.packages: com.example.dto

How do you produce messages with KafkaTemplate?

KafkaTemplate is Spring Kafka's primary abstraction for sending messages, providing both synchronous and asynchronous sending options. The fire-and-forget approach is simplest but provides no feedback on success or failure. Adding callbacks enables logging and error handling for asynchronous sends, while the synchronous approach blocks until the broker acknowledges receipt—useful when you need guaranteed delivery before proceeding.

@Service
@RequiredArgsConstructor
public class OrderProducer {
 
    private final KafkaTemplate<String, Order> kafkaTemplate;
 
    // Fire and forget
    public void sendOrder(Order order) {
        kafkaTemplate.send("orders", order.getId(), order);
    }
 
    // With callback
    public void sendOrderWithCallback(Order order) {
        CompletableFuture<SendResult<String, Order>> future =
            kafkaTemplate.send("orders", order.getId(), order);
 
        future.whenComplete((result, ex) -> {
            if (ex == null) {
                log.info("Sent order {} to partition {} offset {}",
                    order.getId(),
                    result.getRecordMetadata().partition(),
                    result.getRecordMetadata().offset());
            } else {
                log.error("Failed to send order {}", order.getId(), ex);
            }
        });
    }
 
    // Synchronous (blocking)
    public void sendOrderSync(Order order) throws Exception {
        kafkaTemplate.send("orders", order.getId(), order).get(10, TimeUnit.SECONDS);
    }
}

How do you consume messages with @KafkaListener?

The @KafkaListener annotation declaratively configures message consumption, automatically deserializing messages and invoking the annotated method for each record. You can access message metadata through parameter annotations, process batches for higher throughput, or use manual acknowledgment for precise control over offset commits.

Manual acknowledgment lets the application commit after successful handling, which is normally at-least-once. Spring Kafka exactly-once processing uses Kafka transactions and a transaction-aware listener container; external side effects still need their own consistency strategy.

@Service
@Slf4j
public class OrderConsumer {
 
    // Basic listener
    @KafkaListener(topics = "orders", groupId = "order-processor")
    public void processOrder(Order order) {
        log.info("Received order: {}", order.getId());
        // Process order
    }
 
    // With metadata
    @KafkaListener(topics = "orders", groupId = "order-processor")
    public void processOrderWithMetadata(
            @Payload Order order,
            @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
            @Header(KafkaHeaders.OFFSET) long offset,
            @Header(KafkaHeaders.RECEIVED_TIMESTAMP) long timestamp) {
        log.info("Received order {} from partition {} at offset {}",
            order.getId(), partition, offset);
    }
 
    // Batch processing
    @KafkaListener(topics = "orders", groupId = "batch-processor",
                   containerFactory = "batchFactory")
    public void processOrderBatch(List<Order> orders) {
        log.info("Received batch of {} orders", orders.size());
        orders.forEach(this::processOrder);
    }
 
    // Manual acknowledgment
    @KafkaListener(topics = "orders", groupId = "manual-ack")
    public void processWithManualAck(Order order, Acknowledgment ack) {
        try {
            processOrder(order);
            ack.acknowledge();  // Commit offset only after successful processing
        } catch (Exception e) {
            // Don't ack - message will be redelivered
            throw e;
        }
    }
}

How do you handle errors and retries in Spring Kafka?

Spring Kafka provides flexible error handling through DefaultErrorHandler with configurable retry policies and dead letter topic publishing. Failed messages can be automatically retried with backoff delays before being sent to a dead letter topic for manual investigation. The @RetryableTopic annotation simplifies this further by automatically creating retry topics with exponential backoff.

@Configuration
public class KafkaConfig {
 
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Order> kafkaListenerContainerFactory(
            ConsumerFactory<String, Order> consumerFactory) {
 
        ConcurrentKafkaListenerContainerFactory<String, Order> factory =
            new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
 
        // Retry configuration
        factory.setCommonErrorHandler(new DefaultErrorHandler(
            new DeadLetterPublishingRecoverer(kafkaTemplate),
            new FixedBackOff(1000L, 3)  // 3 retries, 1 second apart
        ));
 
        return factory;
    }
}
 
// Or use retry topics (Spring Kafka 2.7+)
@RetryableTopic(
    attempts = "3",
    backoff = @Backoff(delay = 1000, multiplier = 2),
    dltTopicSuffix = ".DLT",
    autoCreateTopics = "true"
)
@KafkaListener(topics = "orders")
public void processWithRetryTopic(Order order) {
    // Failures automatically sent to orders-retry-0, orders-retry-1, then orders.DLT
    processOrder(order);
}
 
@DltHandler
public void handleDlt(Order order) {
    log.error("Message exhausted retries: {}", order);
    // Alert, store for manual review, etc.
}

Replication and Reliability Questions

Production Kafka deployments require careful configuration for reliability and fault tolerance.

How does Kafka replication work?

Kafka replication provides fault tolerance by maintaining multiple copies of each partition across different brokers. Each partition has one leader replica that handles all client reads and writes, while follower replicas passively replicate data from the leader. The set of replicas that are caught up with the leader form the In-Sync Replica set (ISR), which determines which replicas are eligible to become leader if the current leader fails.

Followers that fall behind (due to network issues or slow disks) are removed from the ISR until they catch up, ensuring that leader election only promotes fully synchronized replicas.

flowchart LR
    subgraph B1["Broker 1 (Leader)"]
        L1["Message 1"]
        L2["Message 2"]
        L3["Message 3<br/>(committed)"]
    end
 
    subgraph B2["Broker 2 (Follower)"]
        F1["Message 1"]
        F2["Message 2"]
        F3["Message 3<br/>(in ISR)"]
    end
 
    subgraph B3["Broker 3 (Follower)"]
        R1["Message 1"]
        R2["Message 2"]
        R3["(catching up)<br/>(not in ISR)"]
    end
 
    B2 -->|"replicates"| B1
    B3 -.->|"catching up"| B1

In-Sync Replicas (ISR): Replicas that are:

• Assigned as replicas of the partition
• Fetching from the leader within replica.lag.time.max.ms

What is min.insync.replicas and why is it important?

The min.insync.replicas setting defines the minimum number of replicas that must acknowledge a write when using acks=all. This prevents data loss in scenarios where most replicas are unavailable—if the ISR falls below this threshold, the broker rejects writes with a NOT_ENOUGH_REPLICAS error rather than accepting writes that can't be adequately replicated.

The standard production configuration uses replication factor 3 with min.insync.replicas=2, allowing the cluster to tolerate one broker failure while still accepting writes, and tolerating a second failure for reads.

# Topic configuration
min.insync.replicas=2

With acks=all and min.insync.replicas=2:

• Write succeeds only if at least 2 replicas acknowledge
• If ISR drops below 2, writes are rejected (NOT_ENOUGH_REPLICAS)
• Prevents data loss even if one broker fails

Common configuration for durability:

replication.factor=3
min.insync.replicas=2
acks=all

Performance Tuning Questions

Optimizing Kafka for your specific workload requires understanding the throughput-latency trade-off.

How do you optimize Kafka for high throughput?

High throughput optimization focuses on maximizing messages per second through larger batches, compression, and parallel processing. Larger batch sizes with increased linger time allow producers to accumulate more messages before sending, reducing per-message overhead. Compression (especially LZ4 or zstd) reduces network bandwidth at the cost of CPU. Consumer-side tuning focuses on fetching more data per request and processing larger batches.

These settings increase latency since messages wait longer before being sent or fetched, so they're appropriate for batch processing workloads rather than real-time applications.

Producer tuning:

# Larger batches
batch.size=65536
linger.ms=10
 
# Compression
compression.type=lz4
 
# More in-flight requests (with idempotence)
max.in.flight.requests.per.connection=5
 
# Larger buffer
buffer.memory=67108864

Consumer tuning:

# Fetch more data per request
fetch.min.bytes=1048576
fetch.max.wait.ms=500
 
# Larger fetch size
max.partition.fetch.bytes=1048576
 
# Process in batches
max.poll.records=500

Broker tuning:

# More I/O threads
num.io.threads=16
num.network.threads=8
 
# Socket buffers
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576

How do you optimize Kafka for low latency?

Low latency optimization prioritizes fast message delivery over throughput efficiency. Setting linger.ms=0 sends messages immediately without waiting for batching. Consumer-side, fetch.max.wait.ms=0 ensures brokers return data immediately rather than waiting for more messages to accumulate. These settings reduce the time between a producer sending a message and a consumer receiving it, but result in more network requests and lower overall throughput.

For the lowest latency, also consider using SSD storage on brokers and ensuring adequate network bandwidth.

# Producer: Send immediately
linger.ms=0
batch.size=16384
 
# Consumer: Fetch frequently
fetch.max.wait.ms=0
fetch.min.bytes=1
 
# Broker: Faster flushing
log.flush.interval.messages=1  # Careful: impacts durability

Trade-off: Lower latency typically means lower throughput.

Monitoring and Operations Questions

Operational excellence is key to running Kafka in production.

What metrics should you monitor for Kafka?

Effective Kafka monitoring covers brokers, producers, and consumers. Broker metrics track cluster health—under-replicated partitions indicate replication problems, while offline partitions mean data unavailability. Producer metrics reveal send performance and errors. Consumer metrics, especially lag, show whether consumers are keeping up with the data flow.

Set up alerts on critical metrics like under-replicated partitions, offline partitions, and consumer lag exceeding thresholds to catch problems before they impact applications.

Broker metrics:

• kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec - Message rate
• kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions - Replication health
• kafka.controller:type=KafkaController,name=OfflinePartitionsCount - Availability
• kafka.network:type=RequestMetrics,name=RequestsPerSec - Request rate

Producer metrics:

• record-send-rate - Messages sent per second
• record-error-rate - Failed sends
• request-latency-avg - Average request latency
• batch-size-avg - Average batch size

Consumer metrics:

• records-consumed-rate - Consumption rate
• records-lag-max - Maximum lag across partitions
• commit-latency-avg - Offset commit latency
• rebalance-rate-and-time - Rebalance frequency

How do you detect and troubleshoot consumer lag?

Consumer lag is the difference between the latest offset in a partition and the consumer's committed offset, representing how far behind the consumer is from real-time. Persistent or growing lag indicates consumers can't keep up with the rate of incoming messages. Diagnosing the cause requires examining whether processing is slow, there are too few consumers, or external factors like database writes or network issues are creating bottlenecks.

# Check consumer group lag
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --describe --group my-consumer-group
 
# Output:
# TOPIC    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders   0          1000            1500            500
# orders   1          2000            2100            100

Causes of consumer lag:

1. Slow processing: Optimize consumer logic or increase parallelism
2. Insufficient consumers: Add more consumers (up to partition count)
3. Network issues: Check connectivity and bandwidth
4. Rebalancing storms: Use cooperative rebalancing, increase session.timeout.ms
5. GC pauses: Tune JVM garbage collection

What is Schema Registry and why is it important?

Schema Registry provides centralized schema management for Kafka, storing schemas for message keys and values and enforcing compatibility rules during evolution. Without Schema Registry, producers and consumers must coordinate schema changes manually, risking deserialization failures when schemas don't match. With Schema Registry, schemas are versioned and checked for compatibility before being registered, preventing breaking changes.

The registry supports multiple serialization formats including Avro, Protobuf, and JSON Schema, with Avro being the most common choice due to its compact binary format and strong schema evolution support.

// Producer with Avro and Schema Registry
props.put("schema.registry.url", "http://localhost:8081");
props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
 
// Consumer
props.put("value.deserializer", "io.confluent.kafka.serializers.KafkaAvroDeserializer");
props.put("specific.avro.reader", "true");

Compatibility modes:

• BACKWARD: New schema can read old data
• FORWARD: Old schema can read new data
• FULL: Both backward and forward compatible
• NONE: No compatibility checks

What are common Kafka production issues and their solutions?

Production Kafka clusters encounter predictable categories of issues that operators should be prepared to diagnose and resolve. Under-replicated partitions typically indicate broker problems—check disk I/O, network connectivity, and JVM health. Consumer lag requires analyzing whether the bottleneck is processing speed, consumer count, or external dependencies. Rebalancing storms often result from unstable consumers with heartbeat timeouts; increasing session timeouts and using cooperative rebalancing helps.

Quick Reference

Related Articles

• Microservices Architecture Interview Guide - Kafka in microservices context
• System Design Interview Guide - Distributed systems fundamentals
• Spring Boot Interview Guide - Spring Kafka integration
• Complete Java Backend Developer Interview Guide - Full Java backend preparation

Official Sources

This guide was reviewed against the Apache Kafka documentation and supported releases available on September 2, 2026:

• Apache Kafka downloads - current supported releases, including Kafka 4.3.1
• Apache Kafka 4.0 release announcement - removal of ZooKeeper and GA of the next-generation consumer protocol
• KRaft versus ZooKeeper - KRaft behavior and removed ZooKeeper features
• Consumer Rebalance Protocol - KIP-848 opt-in and server-side settings
• Kafka 4.3 producer configuration - acks, retries, and idempotence defaults
• Kafka design: message delivery semantics - at-most-once, at-least-once, idempotence, and transactions

Frequently Asked Questions

What is Apache Kafka and when should you use it?

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, real-time data pipelines. Use it when you need to decouple services, process millions of events per second, build event-driven architectures, or implement event sourcing. Common use cases include log aggregation, metrics collection, stream processing, and microservices communication.

How do Kafka partitions enable scalability?

Partitions are Kafka's unit of parallelism. Each partition is an ordered, immutable sequence of records that can be consumed independently. By distributing partitions across brokers and assigning different partitions to different consumers in a group, Kafka enables horizontal scaling. More partitions mean more parallel consumers, but ordering is only guaranteed within a single partition.

What are consumer groups and how do they work?

A consumer group is a set of consumers that cooperate to consume messages from topics. Each partition is assigned to exactly one consumer within the group, enabling load balancing. Kafka tracks offsets per consumer group, allowing multiple groups to independently consume the same topic. When consumers join or leave, Kafka triggers a rebalance to redistribute partitions.

How does Kafka guarantee message delivery?

Producer acks controls durability, not end-to-end processing semantics. At-most-once or at-least-once consumption depends mainly on when offsets are committed relative to processing. Idempotence deduplicates producer retries, while transactions make writes to Kafka partitions and consumed offsets atomic; downstream systems outside Kafka still require their own idempotency or transaction strategy.

What is the difference between Kafka Streams and ksqlDB?

Kafka Streams is a Java library for building stream processing applications that run as regular JVM applications. ksqlDB is a streaming SQL engine that runs as a separate server and lets you write stream processing logic using SQL syntax. Use Kafka Streams when you need full programmatic control and want to embed processing in your application. Use ksqlDB for rapid prototyping, simpler transformations, or when SQL is preferred.

How do you monitor Kafka consumer lag?

Consumer lag is the difference between the latest offset in a partition and the consumer's committed offset. Monitor it using: Kafka's built-in consumer-groups command, JMX metrics (records-lag-max), tools like Burrow or Kafka Lag Exporter, or cloud provider dashboards. High lag indicates consumers can't keep up with producers. Set alerts on lag thresholds and lag growth rate.