kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Randall Hauch (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6738) Kafka Connect handling of bad data
Date Mon, 02 Apr 2018 15:06:00 GMT
Randall Hauch created KAFKA-6738:

             Summary: Kafka Connect handling of bad data
                 Key: KAFKA-6738
                 URL: https://issues.apache.org/jira/browse/KAFKA-6738
             Project: Kafka
          Issue Type: Improvement
          Components: KafkaConnect
    Affects Versions: 1.1.0
            Reporter: Randall Hauch
            Assignee: Konstantine Karantasis

Kafka Connect connectors and tasks fail when they run into an unexpected situation or error,
but the framework should provide more general "bad data handling" options, including (perhaps
among others):

# fail fast, which is what we do today (assuming connector actually fails and doesn't eat
# retry (possibly with configs to limit)
# drop data and move on
# dead letter queue

This needs to be addressed in a way that handles errors from:

# The connector itself (e.g. connectivity issues to the other system)
# Converters/serializers (bad data, unexpected format, etc)
# SMTs
# Ideally the framework as well, though we obviously want to fix known bugs anyway

This message was sent by Atlassian JIRA

View raw message