Tag: kafka

TIL #2: Kafka poison pill message and CommitFailedException

Yesterday I was working with a team that was facing issue with their Kafka related code. The Kafka consumer was failing with the exception

[] ERROR [2022-11-22 08:32:52,853] com.abc.MyKakfaConsumer: Exception while processing events
! java.lang.NullPointerException: Cannot invoke "org.apache.kafka.common.header.Header.value()" because the return value of "org.apache.kafka.common.header.Headers.lastHeader(String)" is null
! at com.abc.MyKakfaConsumer.run(MyKakfaConsumer.java:83)
! at java.base/java.lang.Thread.run(Thread.java:833)

The consumer code looked like as shown below.

Continue reading “TIL #2: Kafka poison pill message and CommitFailedException”

The 5 minute introduction to Log-Based Change Data Capture with Debezium

Few years back I was working on an application where I had to pull data from an event table(populated using database triggers) and update my application specific data stores . This is a common problem that most software web developers need to solve. At that time I was not aware that this problem has a name. Sometime later I learnt that this is called Change Data Capture (CDC). As per wikipedia article on change data capture,

In databases, change data capture (CDC) is a set of software design patterns used to determine (and track) the data that has changed so that action can be taken using the changed data.

The key benefit of CDC is that you can identify the changed data in your source database which you can then incrementally apply to your target system. In absence of CDC, we are left with doing bulk loading of the data which is both time consuming and costly.

Continue reading “The 5 minute introduction to Log-Based Change Data Capture with Debezium”