kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srinivas, Kaushik (Nokia - IN/Bangalore)" <kaushik.srini...@nokia.com>
Subject kafka connect stops consuming data when kafka broker goes down
Date Mon, 03 Jun 2019 11:52:13 GMT
Hello kafka dev,

We are encountering an issue when kafka connect is running hdfs sink connector pulling data
from kafka and writing to hdfs location.
In between when the data is flowing in the pipeline from producer -> kafka topic ->
kafka connect hdfs sink connector -> hdfs,
If even one of the kafka broker goes down, the connect framework stops responding. Stops consuming
records and REST API also becomes not interactive.

Until the kafka connect framework is restarted, it would not pull the data from kafka and
REST api remains inactive. Nothing is coming in the logs as well.
Checked the topics in kafka used by connect, everything has been reassigned to another broker
and has the leader available.

Has anyone encountered this issue ? what would be the expected behavior ?

Thanks in advance
Kaushik

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message