Outbox pattern with SpringBoot Kotlin and Kafka

In a modern microservices architecture, services must communicate asynchronously, decoupled, and reliably. Apache Kafka has become a standard for event transport, but naive usage hides a major risk.

👉 How can we ensure that a Kafka event always represents a state that has actually been persisted in the database ?

This problem is very common in production and often causes data inconsistencies across services.

In this article, we will tell the full story of this problem, and build a robust solution step by step : the Outbox Pattern with Spring Boot Kotlin and Kafka.

Our goal is clear :

• understand why before how

• introduce code only when necessary

• achieve a complete, realistic, and production-ready implementation

1. Context and global architecture

Before writing any code, let's define the architectural context.

Our system consists of :

• a Spring Boot (Kotlin) business microservice

• a Transactional database (PostgreSQL)

• Apache Kafka as the event bus

Core microservices principle :

Each service owns its data and its events.

Kafka does not store business state. It transports facts that have already been validated.

2. First attempt : Naive implementation

Many teams start with a simple approach.

@Service
class CommandService(
    private val commandRepository: CommandRepository,
    private val kafkaTemplate: KafkaTemplate<String, String>
) {

    fun createCommand(cmd: Command) {
        commandRepository.save(cmd)
        kafkaTemplate.send("commands", serialize(cmd))
    }
}

At first glance, the flow seems logical :

1. save the command

2. publish the Kafka event

But let's look at a critical failure scenario.

3. The failure scenario that breaks everything

👉 What happens if the database transaction succeeds but Kafka is unavailable ?

Result :

• the command exists in the database

• the event was never published

The system becomes permanently inconsistent.

This is not an isolated bug.
It is a structural problem.

4. Why this is an architectural problem

This behavior is inevitable because :

• the database is transactional

• Kafka does not participate in this transaction

• there is no atomic guarantee between the two

Distributed transactions (XA, 2PC) are neither practical nor recommended with Kafka.

👉 We need to change the mental model.

5. Paradigm shift : An event is data

The outbox pattern is based on a key idea :

A business event is a piece of business data.

If it is critical, it must be persisted transactionally.

Instead of publishing directly to Kafka, we will :

1. persist the event

2. in the same transaction as the business data

This gives birth to the outbox table.

6. Modeling the outbox table

Add a dedicated table in the service's database.

@Table("outbox_event")
data class OutboxEvent(
    @Id val id: Long? = null,
    val aggregateType: String,
    val aggregateId: String,
    val eventType: String,
    val payload: String,
    val createdAt: LocalDateTime = LocalDateTime.now()
)

At this stage :

• no Kafka yet

• no asynchronous behavior

• only reliable persistence

7. Atomic write : Business data + Event

Modify the business service accordingly.

@Service
class CommandService(
    private val commandRepository: CommandRepository,
    private val outboxRepository: OutboxRepository
) {

    @Transactional
    fun createCommand(cmd: Command): Mono<Void> {
        return commandRepository.save(cmd)
            .then(outboxRepository.save(
                OutboxEvent(
                    aggregateType = "Command",
                    aggregateId = cmd.id,
                    eventType = "CommandCreated",
                    payload = serialize(cmd)
                )
            ))
            .then()
    }
}

Now :

• if the transaction fails → nothing is written

• if it succeeds → the state and the event exist together

The main risk is eliminated.

8. Publishing events : Polling and its limits

One simple approach is to poll the outbox table periodically.

@Component
class OutboxPollingPublisher(
    private val outboxRepository: OutboxRepository,
    private val kafkaTemplate: KafkaTemplate<String, String>
) {

    @Scheduled(fixedDelay = 5000)
    fun publish() {
        outboxRepository.findUnpublished()
            .forEach { event ->
                kafkaTemplate.send("commands", event.payload)
                outboxRepository.markAsPublished(event.id!!)
            }
    }
}

This works but quickly has problems :

• heavy load on the database

• complex retry logic

• risk of duplicates in case of crashes

• latency depends on polling frequency

👉 It does not scale well.

9. CDC : Listening to the database instead of the application

Change Data Capture (CDC) offers a radically different approach.

The database already knows exactly what has been committed.

Instead of adding application code, we rely on :

• the database transaction log (WAL / binlog)

Every committed INSERT becomes an event.

Key advantage:

• no Kafka logic in the application

10. Debezium : Implementing CDC with Kafka

Debezium is a Kafka Connect component that :

• reads database logs

• captures changes

• automatically publishes to Kafka

Typical configuration for the Outbox Pattern:

{
  "name": "outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "user",
    "database.password": "password",
    "database.dbname": "orders",
    "table.include.list": "public.outbox_event",
    "plugin.name": "pgoutput",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.route.by.field": "aggregate_type"
  }
}

Result :

• every insertion into outbox_event

• becomes a Kafka message

• without any direct Kafka dependency in the application

11. Final architecture of the outbox pattern

        Client
          │
    Spring Boot Service
          │  (transaction)
        Database
          │  (WAL / CDC)
        Debezium
          │
        Kafka

Responsibilities :

• the service writes consistent data

• the database guarantees atomicity

• Debezium captures committed changes

• Kafka distributes events

12. Production benefits

With the Outbox Pattern :

• no events are lost

• safe restarts

• strong decoupling from Kafka

• improved observability

• resilient architecture

Conclusion

The outbox pattern is not a trick.
It is a foundational architectural approach for distributed systems.

Events are data.

By relying on the database as the source of truth and Kafka as the delivery mechanism, we build reliable, scalable, and maintainable systems.

In the next article, we will cover how to complement this approach on the consumer side with the Inbox Pattern for end-to-end reliability.