Question 1

What is Apache Kafka and what problems does it solve?

Accepted Answer

Apache Kafka is a distributed event streaming platform that enables high-throughput, fault-tolerant, real-time data pipelines and streaming applications. It solves problems related to handling massive volumes of real-time data streams, decoupling data producers from consumers, and providing durable, scalable message storage with replay capabilities.

Question 2

What are the core components of Kafka architecture?

Accepted Answer

Kafka's core components include brokers (servers that store and serve data), topics (logical channels for organizing messages), partitions (subdivisions of topics for parallelism), producers (applications that publish messages), and consumers (applications that read messages). These components work together to provide a distributed, fault-tolerant streaming platform.

Question 3

What are partitions in Kafka and why are they important?

Accepted Answer

Partitions are ordered, immutable sequences of messages that subdivide Kafka topics, enabling horizontal scalability, parallelism, and fault tolerance. Each partition is replicated across multiple brokers, and messages within a partition maintain strict ordering, making partitions the fundamental unit of distribution and parallelism in Kafka.

Question 4

What is the role of offsets in Kafka?

Accepted Answer

An offset is a unique sequential integer assigned to each message within a Kafka partition, starting from 0. Offsets serve as the position pointer for consumers, enabling them to track which messages they've processed, resume from failure points, and replay historical data by resetting to earlier offsets.

Question 5

What is the role of ZooKeeper in Kafka and what is KRaft mode?

Accepted Answer

ZooKeeper historically served as Kafka's metadata store and coordination service, managing broker membership, leader elections, and configuration. KRaft mode is the new ZooKeeper-free architecture that uses an internal Raft consensus protocol, simplifying deployment and improving scalability by eliminating the external ZooKeeper dependency.

Question 6

What is a Kafka topic?

Accepted Answer

A Kafka topic is a logical channel or category where messages are published and stored. Topics provide logical separation of different data streams and can be subdivided into multiple partitions for scalability and parallelism. Each topic has a name and configurable retention policies.

Question 7

What is a Kafka producer and how does it work?

Accepted Answer

A Kafka producer is a client application that publishes records to Kafka topics. It serializes data, determines the partition using key hashing or custom logic, buffers records in memory for batching, and sends them to the appropriate broker leader, handling retries and acknowledgments automatically.

Question 8

What are the producer acknowledgment settings (acks)?

Accepted Answer

The acks setting controls write durability: acks=0 sends without waiting (fastest, may lose data), acks=1 waits for leader acknowledgment (balanced), and acks=all waits for all in-sync replicas (most durable, slowest). This represents a throughput versus durability trade-off.

Question 9

What is a consumer group and why is it important?

Accepted Answer

A consumer group is a set of consumers that cooperate to consume messages from Kafka topics, where each partition is consumed by exactly one consumer within the group. This enables horizontal scalability and load balancing, allowing multiple consumers to process messages in parallel while maintaining ordering guarantees within each partition.

Question 10

What is consumer rebalancing?

Accepted Answer

Consumer rebalancing is the process where Kafka redistributes partition assignments among consumers in a group when membership changes or when topic partitions are added. During rebalancing, consumers temporarily stop consuming messages while partitions are reassigned, which can cause brief processing delays.

Question 11

What are In-Sync Replicas (ISR)?

Accepted Answer

In-Sync Replicas (ISR) are replicas that have caught up with the leader partition and are eligible to become the new leader if the current leader fails. A replica is in the ISR if it has fully fetched the leader's log within the replica.lag.time.max.ms window, ensuring data durability and availability.

Question 12

What is the replication factor in Kafka?

Accepted Answer

The replication factor defines how many copies of each partition exist across the Kafka cluster. A replication factor of 3 is the industry standard, providing a good balance between fault tolerance (can survive 2 broker failures) and resource usage.

Question 13

What is log compaction in Kafka?

Accepted Answer

Log compaction is a retention policy that keeps only the most recent value for each message key, rather than deleting old messages after a time or size limit. It's ideal for changelog topics, stateful applications, and snapshots where you need the latest state for each key but don't care about historical updates.

Question 14

How does Kafka handle broker failures?

Accepted Answer

When a broker fails, Kafka automatically elects new leaders for all partitions where the failed broker was the leader, promoting one of the in-sync replicas. Clients experience brief interruptions as they discover the new leaders through metadata refreshes, with the entire failover process typically completing in under 30 seconds.

Question 15

What is the role of the message key in Kafka?

Accepted Answer

The message key serves two primary purposes: partitioning and message ordering. Kafka uses the key's hash to determine which partition receives the message, ensuring all messages with the same key go to the same partition and are therefore ordered relative to each other.

Kafka Fundamentals(6 questions)

What is Apache Kafka and what problems does it solve?

What are the core components of Kafka architecture (brokers, topics, partitions, producers, consumers)?

How does Kafka differ from traditional message queues like RabbitMQ or ActiveMQ?

What are partitions in Kafka and why are they important?

Explain the concept of offsets in Kafka.

What is the role of ZooKeeper in Kafka and what is KRaft mode?

Producers(5 questions)

What is a Kafka producer and how does it work?

What is the role of the key in Kafka messages?

What are the producer acknowledgment settings (acks=0, acks=1, acks=all)?

How do you handle producer retries and idempotence?

What are producer batching and linger.ms settings?

Consumers(6 questions)

What is a consumer group and why is it important?

What is consumer rebalancing and how does it work?

How do you commit offsets in Kafka?

What is the difference between at-least-once, at-most-once, and exactly-once semantics?

How do you handle consumer lag?

What is a consumer heartbeat and session timeout?

Topics and Partitions(5 questions)

What factors should you consider when choosing the number of partitions?

What is message ordering guarantee in Kafka?

What are In-Sync Replicas (ISR) and why are they important?

What is log compaction and when would you use it?

What are topic configurations (retention.ms, retention.bytes, segment.bytes)?

Replication and Fault Tolerance(5 questions)

How does Kafka replication work?

What is the replication factor and how do you choose it?

What happens during a broker failure?

What is unclean leader election and should you enable it?

What is the high watermark in Kafka?

Kafka Connect(4 questions)

What is Kafka Connect and what problems does it solve?

What is the difference between source and sink connectors?

What is Single Message Transform (SMT) in Kafka Connect?

What is the Dead Letter Queue (DLQ) pattern in Kafka Connect?

Kafka Streams(4 questions)

What is Kafka Streams and how does it differ from other stream processing frameworks?

What is the difference between KStream and KTable?

What are state stores in Kafka Streams?

What is the difference between tumbling, hopping, sliding, and session windows?

Schema Registry(3 questions)

What is Schema Registry and why is it important?

What is schema compatibility and what types exist?

What is the difference between backward, forward, and full compatibility?

Security(3 questions)

What are the security features available in Kafka?

How do you configure authentication in Kafka (SASL/PLAIN, SASL/SCRAM, SSL)?

What is Kafka authorization and how do ACLs work?

Performance and Optimization(4 questions)

What are the key factors affecting Kafka performance?

How do you tune Kafka for high throughput?

How do you configure compression in Kafka?

What is the impact of replication on performance?

Monitoring and Operations(4 questions)

What are the critical Kafka metrics to alert on?

How do you monitor consumer lag?

What causes under-replicated partitions and how do you fix them?

What is MirrorMaker and how do you use it for replication?

Use Cases and Patterns(4 questions)

What are the common use cases for Kafka?

How do you implement event sourcing with Kafka?

How do you implement change data capture (CDC) with Kafka?

What is event-driven architecture and how does Kafka enable it?

Transactions(3 questions)

What are Kafka transactions and when would you use them?

What is read_committed vs read_uncommitted isolation level?

How do you implement exactly-once semantics with transactions?

Troubleshooting and Common Issues(4 questions)

How do you diagnose and fix consumer group rebalancing issues?

What causes message duplication and how do you prevent it?

What causes high consumer lag and how do you address it?

What are the common causes of data loss in Kafka and how do you prevent them?