Most engineers think Kafka gives you exactly-once out of the box. It doesn't. Here's what actually happens under the hood — and where it silently breaks.

The Delivery Semantics Spectrum

Before we go deep on Kafka, let's be precise about what we mean. In distributed messaging, there are exactly three delivery guarantees:

At-Most-Once — Message delivered 0 or 1 times → Risk: Data loss

At-Least-Once — Message delivered 1 or more times → Risk: Duplicates

Exactly-Once — Message delivered precisely once → Risk: Complex, not free

Every distributed system lives somewhere on this spectrum. The question is never "which is best" — it's "which can I afford for this use case."

A payment system cannot tolerate duplicates. A log aggregator can. An inventory deduction system cannot. A notification system probably can.

Kafka supports all three semantics — but you have to configure for them explicitly. The default is at-least-once. Exactly-once requires understanding what's actually happening under the hood.

How Kafka Works Internally

To understand delivery guarantees, you need a mental model of what Kafka actually is.

Producer → Topic (Partitions) → Consumer Group

A Topic is a logical stream of records. Each topic is split into Partitions — ordered, immutable logs. Each partition lives on a Broker (a Kafka server node).

Topic: "orders"
├── Partition 0  → Broker 1 (Leader), Broker 2 (Replica)
├── Partition 1  → Broker 2 (Leader), Broker 3 (Replica)
└── Partition 2  → Broker 3 (Leader), Broker 1 (Replica)

Producers write to a partition's leader. Consumers read from partitions and track their position using an offset — a simple integer that says "I've read up to message #47."

This offset is the key to everything that follows.

Partition 0:
[msg@0] [msg@1] [msg@2] [msg@3] [msg@4] ...
                              ↑
                     Consumer offset (committed)

When a consumer commits offset 3, it's saying: "I've successfully processed messages 0–3. Next time I start, begin at 4."

Where and when this commit happens determines your delivery semantic.

At-Most-Once Delivery

Configuration: Commit offset before processing the message.

// Dangerous pattern — at-most-once
consumer.run({
  eachMessage: async ({ message, heartbeat, commitOffsets }) => {
    // Commit FIRST
    await commitOffsets();

    // Now process — if this crashes, message is LOST
    await processOrder(message.value);
  },
});

If the process crashes after the commit but before processOrder, the message is gone forever. The consumer restarts at the next offset.

When to use: Metrics, telemetry, logs — where occasional data loss is acceptable and throughput matters more than correctness.

At-Least-Once Delivery

Configuration: Commit offset after processing. This is Kafka's default behavior.

// Default Kafka behavior — at-least-once
const consumer = kafka.consumer({ groupId: 'order-service' });

await consumer.run({
  autoCommit: true, // commits after eachMessage resolves
  eachMessage: async ({ topic, partition, message }) => {
    // Process first
    await processOrder(message.value);

    // Offset committed automatically after this resolves
    // BUT: if we crash mid-processing and restart,
    // we'll re-process the same message
  },
});

The problem: if your consumer crashes during processing but after doing some side effects (writing to DB, calling an API), and then restarts — it will re-process the same message. You get duplicates.

Timeline:
1. Consumer reads message @ offset 5
2. Consumer writes to database ✅
3. Consumer calls payment API ✅
4. Consumer CRASHES before committing offset ❌
5. Consumer restarts, reads message @ offset 5 again
6. Writes to database AGAIN 💀
7. Calls payment API AGAIN 💀 (duplicate charge!)

When to use: Most use cases where your consumers are idempotent — i.e., processing the same message twice produces the same result. This is the most common real-world setup.

The Exactly-Once Problem

Exactly-once is hard because distributed systems have two independent failure modes:

  1. Producer → Broker: The producer sends a message, the broker writes it, but the ACK is lost in the network. The producer retries and sends a duplicate.

  2. Consumer → Processing: The consumer processes a message and crashes before committing the offset. On restart, it reprocesses.

Producer                Broker
   |---[send msg]------→|
   |                    | (writes to log)
   |←--[ACK]---X (lost)|
   |                    |
   |---[retry msg]-----→| ← DUPLICATE written
   |←--[ACK]------------|

Solving problem #1 requires Idempotent Producers. Solving problem #2 in a multi-system context (Kafka + external DB) is fundamentally impossible without additional mechanisms (we'll get to this).

Solving both within Kafka itself requires Transactions.

Idempotent Producers

Introduced in Kafka 0.11, idempotent producers solve the duplicate-write-on-retry problem at the broker level.

How it works:

Each producer is assigned a Producer ID (PID) by the broker. Every message batch is tagged with:

  • The PID
  • A monotonically increasing sequence number
Message Batch:
{
  producerId: 42,
  epoch: 1,
  sequenceNumber: 7,
  records: [...]
}

The broker maintains a map of (PID, partition) → lastSequenceNumber. If a batch arrives with a sequence number the broker has already seen, it deduplicates silently and returns success to the producer without writing again.

Producer sends batch with seq=7 → Broker writes, ACK lost
Producer retries batch with seq=7 → Broker sees seq=7 already stored
                                  → Returns success, does NOT write again ✅

Enabling idempotent producer:

// KafkaJS
const producer = kafka.producer({
  idempotent: true, // enables exactly-once at producer level
});

// Java (Spring Kafka)
@Bean
public ProducerFactory<String, String> producerFactory() {
    Map<String, Object> config = new HashMap<>();
    config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
    // These are automatically set when idempotence is enabled:
    // acks = all
    // retries = Integer.MAX_VALUE
    // max.in.flight.requests.per.connection = 5
    return new DefaultKafkaProducerFactory<>(config);
}

Important: Idempotent producers only prevent duplicates within a single producer session and single partition. A producer restart gets a new PID, which means the deduplication window resets.

Kafka Transactions

Idempotent producers solve producer-side duplicates. But what if you need to:

  • Read from topic A, process, write to topic B — atomically
  • Write to multiple partitions — atomically
  • Ensure consumers only see committed messages

This is where Kafka Transactions come in.

The Transaction API

// KafkaJS — Transactional producer
const producer = kafka.producer({
  transactionalId: 'order-processor-1', // unique, stable ID
  idempotent: true,                      // required for transactions
  maxInFlightRequests: 5,
});

await producer.connect();

const transaction = await producer.transaction();

try {
  // Atomically send to multiple topics/partitions
  await transaction.send({
    topic: 'processed-orders',
    messages: [{ key: 'order-123', value: JSON.stringify(order) }],
  });

  await transaction.send({
    topic: 'order-audit-log',
    messages: [{ key: 'order-123', value: JSON.stringify(auditEntry) }],
  });

  await transaction.commit();
} catch (err) {
  await transaction.abort();
  throw err;
}

Read-Process-Write: The Full Exactly-Once Pattern

This is the crown jewel of Kafka's exactly-once story. The idea: commit the consumer offset and produce output in the same transaction.

// Full exactly-once: consume → process → produce (atomically)

const consumer = kafka.consumer({ groupId: 'payment-processor' });
const producer = kafka.producer({
  transactionalId: 'payment-processor-txn-1',
  idempotent: true,
});

await consumer.connect();
await producer.connect();

await consumer.subscribe({ topic: 'raw-payments' });

await consumer.run({
  autoCommit: false, // CRITICAL: we manage commits manually
  eachBatch: async ({ batch, resolveOffset, heartbeat }) => {
    for (const message of batch.messages) {
      const transaction = await producer.transaction();

      try {
        const payment = JSON.parse(message.value.toString());
        const result = await processPayment(payment);

        // Write output to another topic
        await transaction.send({
          topic: 'processed-payments',
          messages: [{ key: payment.id, value: JSON.stringify(result) }],
        });

        // Commit the consumer offset INSIDE the transaction
        // This is the key — offset commit and message produce are atomic
        await transaction.sendOffsets({
          consumerGroupId: 'payment-processor',
          topics: [
            {
              topic: batch.topic,
              partitions: [
                {
                  partition: batch.partition,
                  offset: (parseInt(message.offset) + 1).toString(),
                },
              ],
            },
          ],
        });

        await transaction.commit();
        resolveOffset(message.offset);
        await heartbeat();
      } catch (err) {
        await transaction.abort();
        throw err; // will trigger retry
      }
    }
  },
});

What Makes This Atomic?

The broker uses a Transaction Coordinator (a special partition: __transaction_state) to track transaction state:

States:
Empty → Ongoing → PrepareCommit → CompleteCommit
                → PrepareAbort  → CompleteAbort

When transaction.commit() is called:

  1. Producer sends EndTransaction(commit=true) to the Transaction Coordinator
  2. Coordinator writes PrepareCommit to its log
  3. Coordinator sends WriteTxnMarker to all involved partitions/brokers
  4. Brokers write the commit marker to their partition logs
  5. Coordinator writes CompleteCommit

Consumers with isolation.level=read_committed only see messages up to the last commit marker — they never see in-progress transaction data.

// Consumer must be configured to read only committed data
const consumer = kafka.consumer({
  groupId: 'downstream-service',
  // KafkaJS exposes this via readUncommitted flag
});

// In raw Kafka config:
// isolation.level = read_committed (default is read_uncommitted)

Exactly-Once in Kafka Streams

If you're using Kafka Streams (Java), exactly-once is much simpler to configure:

// application.properties (Spring Boot + Kafka Streams)
spring:
  kafka:
    streams:
      properties:
        processing.guarantee: exactly_once_v2  # Use v2 (Kafka 2.5+)
        # exactly_once_v2 is more efficient than exactly_once
        # Uses one transaction per partition instead of one per task

@Configuration
@EnableKafkaStreams
public class KafkaStreamsConfig {

    @Bean(name = KafkaStreamsDefaultConfiguration.DEFAULT_STREAMS_CONFIG_BEAN_NAME)
    public KafkaStreamsConfiguration streamsConfig() {
        Map<String, Object> props = new HashMap<>();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "payment-stream-processor");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
                  StreamsConfig.EXACTLY_ONCE_V2); // ← this is all you need
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        return new KafkaStreamsConfiguration(props);
    }
}

// Your stream topology — unchanged
@Bean
public KStream<String, Payment> paymentStream(StreamsBuilder builder) {
    KStream<String, Payment> payments = builder.stream("raw-payments");

    payments
        .filter((key, payment) -> payment.getAmount() > 0)
        .mapValues(payment -> enrichPayment(payment))
        .to("processed-payments");

    return payments;
}

With exactly_once_v2, Kafka Streams handles the transactional produce + offset commit automatically for every record processed through the topology.

When Exactly-Once Breaks Down

This is the section most blog posts skip. Here's the truth:

1. External Side Effects Are Not Covered

Kafka's exactly-once guarantee only covers Kafka-to-Kafka operations. The moment you touch an external system, all bets are off.

// This is NOT exactly-once end-to-end
const transaction = await producer.transaction();

try {
  const order = JSON.parse(message.value.toString());

  // This database write is OUTSIDE Kafka's transaction
  await db.query('INSERT INTO orders VALUES ($1)', [order]);
  // ↑ If this succeeds but the Kafka transaction fails/retries,
  //   you've written to DB but not committed the Kafka offset.
  //   On retry, you'll write to DB again = DUPLICATE

  await transaction.send({ topic: 'order-events', messages: [...] });
  await transaction.sendOffsets({ ... });
  await transaction.commit();
} catch (err) {
  await transaction.abort();
  // DB write already happened — you can't roll it back from here
}

Solution: Outbox Pattern

Write to DB and a Kafka outbox table in the same DB transaction, then have a separate process publish from the outbox to Kafka.

-- Both in one DB transaction
BEGIN;
  INSERT INTO orders (id, data) VALUES ('order-123', '...');
  INSERT INTO kafka_outbox (topic, key, value, created_at)
    VALUES ('order-events', 'order-123', '...', NOW());
COMMIT;

// Separate outbox publisher process
async function publishOutbox() {
  const unpublished = await db.query(
    'SELECT * FROM kafka_outbox WHERE published = false ORDER BY created_at LIMIT 100'
  );

  const transaction = await producer.transaction();
  try {
    for (const row of unpublished.rows) {
      await transaction.send({
        topic: row.topic,
        messages: [{ key: row.key, value: row.value }],
      });
    }
    await transaction.commit();

    // Mark as published only after Kafka commit
    const ids = unpublished.rows.map(r => r.id);
    await db.query('UPDATE kafka_outbox SET published = true WHERE id = ANY($1)', [ids]);
  } catch (err) {
    await transaction.abort();
  }
}

2. Producer Restarts Reset Idempotence

Idempotent deduplication is scoped to a single producer session (PID). When a producer restarts:

Old producer: PID=42, sends seq=0,1,2,3
Producer crashes and restarts
New producer: PID=99, starts seq=0,1,2,3

If old seq=3 was in-flight during crash:
→ New producer re-sends it as PID=99/seq=0
→ Broker sees new PID, writes it again = DUPLICATE

Mitigation: Use transactionalId. The broker ties the transactional ID to a single producer at a time. A restarting producer with the same transactionalId will fence the old one (epoch mechanism):

// Always use transactionalId for long-running producers
const producer = kafka.producer({
  transactionalId: 'payment-service-producer', // stable across restarts
  idempotent: true,
});

3. Zombie Producers

With transactions and transactionalId, Kafka uses an epoch to detect zombies. When a producer restarts with the same transactionalId, it gets a new epoch, and the broker rejects any further writes from the old (lower epoch) producer.

Producer A starts with transactionalId="svc-1", epoch=1
Producer A hangs (GC pause, network partition)
Producer B starts with transactionalId="svc-1", epoch=2
Producer A recovers and tries to write → REJECTED (epoch=1 < 2) ✅

This is correct behavior, but it means you should never run two instances with the same transactionalId. Each instance needs a unique one:

// For horizontally scaled services — make transactionalId unique per instance
const transactionalId = `payment-service-${process.env.POD_NAME}-${process.env.PARTITION_ID}`;

4. max.in.flight.requests.per.connection > 5

With idempotent producers, Kafka can still guarantee ordering if max.in.flight.requests.per.connection <= 5. Beyond 5, the deduplication sequence number check can break under retries.

// Safe configuration for idempotent producer
const producer = kafka.producer({
  idempotent: true,
  maxInFlightRequests: 5, // must be <= 5
});

5. Consumer isolation.level Misconfiguration

Even if your producer uses transactions, a consumer with read_uncommitted (the default in many clients) will read in-progress and aborted transactions:

// WRONG — reads uncommitted/aborted data
const consumer = kafka.consumer({ groupId: 'my-service' });

// This is the default in most clients and will expose dirty reads
// If a transaction is aborted, the consumer still processes those messages

In KafkaJS, you need to pass the raw Kafka config:

const kafka = new Kafka({
  brokers: ['localhost:9092'],
  // KafkaJS doesn't expose isolation.level directly yet
  // Use the underlying node-rdkafka or configure at broker/topic level
});

In Java/Spring Kafka this is explicit:

@Bean
public ConsumerFactory<String, String> consumerFactory() {
    Map<String, Object> config = new HashMap<>();
    config.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed"); // ← critical
    config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    config.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-consumer");
    return new DefaultKafkaConsumerFactory<>(config);
}

Real-World Code: NestJS + KafkaJS

Here's a production-style NestJS module wiring exactly-once with the transactional pattern:

// kafka.module.ts
import { Module } from '@nestjs/common';
import { KafkaService } from './kafka.service';

@Module({
  providers: [KafkaService],
  exports: [KafkaService],
})
export class KafkaModule {}

// kafka.service.ts
import { Injectable, OnModuleInit, OnModuleDestroy, Logger } from '@nestjs/common';
import { Kafka, Producer, Consumer, Transaction } from 'kafkajs';

@Injectable()
export class KafkaService implements OnModuleInit, OnModuleDestroy {
  private readonly logger = new Logger(KafkaService.name);

  private kafka: Kafka;
  private producer: Producer;
  private consumer: Consumer;

  constructor() {
    this.kafka = new Kafka({
      clientId: 'payment-service',
      brokers: process.env.KAFKA_BROKERS?.split(',') ?? ['localhost:9092'],
    });

    this.producer = this.kafka.producer({
      transactionalId: `payment-service-${process.env.HOSTNAME}`,
      idempotent: true,
      maxInFlightRequests: 5,
    });

    this.consumer = this.kafka.consumer({
      groupId: 'payment-processor-group',
    });
  }

  async onModuleInit() {
    await this.producer.connect();
    await this.consumer.connect();
    this.logger.log('Kafka producer and consumer connected');
  }

  async onModuleDestroy() {
    await this.producer.disconnect();
    await this.consumer.disconnect();
  }

  async processWithExactlyOnce(
    sourceTopic: string,
    targetTopic: string,
    handler: (value: string) => Promise<string>,
  ): Promise<void> {
    await this.consumer.subscribe({ topic: sourceTopic, fromBeginning: false });

    await this.consumer.run({
      autoCommit: false,
      eachBatch: async ({ batch, resolveOffset, heartbeat, isRunning, isStale }) => {
        for (const message of batch.messages) {
          if (!isRunning() || isStale()) break;

          const transaction: Transaction = await this.producer.transaction();

          try {
            const inputValue = message.value?.toString() ?? '';
            const outputValue = await handler(inputValue);

            await transaction.send({
              topic: targetTopic,
              messages: [
                {
                  key: message.key,
                  value: outputValue,
                  headers: message.headers,
                },
              ],
            });

            await transaction.sendOffsets({
              consumerGroupId: 'payment-processor-group',
              topics: [
                {
                  topic: batch.topic,
                  partitions: [
                    {
                      partition: batch.partition,
                      offset: (Number(message.offset) + 1).toString(),
                    },
                  ],
                },
              ],
            });

            await transaction.commit();
            resolveOffset(message.offset);
            await heartbeat();

            this.logger.log(`Processed message at offset ${message.offset}`);
          } catch (err) {
            this.logger.error(`Transaction failed, aborting: ${err.message}`);
            await transaction.abort();
            throw err;
          }
        }
      },
    });
  }
}

// payment.service.ts
import { Injectable } from '@nestjs/common';
import { KafkaService } from '../kafka/kafka.service';

@Injectable()
export class PaymentService {
  constructor(private readonly kafkaService: KafkaService) {}

  async startProcessing(): Promise<void> {
    await this.kafkaService.processWithExactlyOnce(
      'raw-payments',
      'processed-payments',
      async (rawValue: string) => {
        const payment = JSON.parse(rawValue);

        // Your pure processing logic here
        // WARNING: any external I/O here is NOT covered by exactly-once
        const enriched = {
          ...payment,
          processedAt: new Date().toISOString(),
          status: 'processed',
        };

        return JSON.stringify(enriched);
      },
    );
  }
}

Checklist: Are You Actually Getting Exactly-Once?

Use this before shipping any Kafka-based system where correctness matters:

Producer Side:
☐ idempotent: true is set on the producer
☐ transactionalId is set and unique per producer instance
☐ maxInFlightRequests <= 5
☐ acks = 'all' (auto-set when idempotent=true, but verify)

Consumer Side:
☐ autoCommit: false — offsets committed manually inside transactions
☐ isolation.level = read_committed (check your client's default)
☐ Offsets committed via transaction.sendOffsets(), not consumer.commitOffsets()

System Design:
☐ External side effects (DB, HTTP calls) are handled via Outbox Pattern
☐ No two producer instances share the same transactionalId
☐ Downstream consumers are idempotent as a safety net
☐ Dead letter topics configured for poison pill messages

Operations:
☐ Transaction timeout configured (default 60s — tune to your processing latency)
☐ Broker-side: transaction.state.log.replication.factor >= 3 in production
☐ Monitoring: transaction abort rate tracked as a key metric

Summary

Kafka's exactly-once guarantee is real — but it's scoped, conditional, and requires deliberate configuration. Here's the mental model to keep:

Exactly-once WITHIN Kafka:
  ✅ Idempotent producer → no duplicate writes on retry
  ✅ Transactions → atomic read-process-write across topics
  ✅ read_committed → consumers see clean data only

Exactly-once ACROSS systems:
  ❌ Not possible natively
  ✅ Outbox Pattern → DB + Kafka consistency
  ✅ Idempotent consumers → safety net for duplicates

The engineers who truly understand this — who know not just that Kafka has transactions, but why idempotence alone isn't enough, and where the guarantee ends — are the ones interviewers at high-scale companies are genuinely looking for.

Build the mental model. Then build the system.