kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanna Subburaj (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6490) JSON SerializationException Stops Connect
Date Wed, 07 Feb 2018 03:36:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354908#comment-16354908

Prasanna Subburaj commented on KAFKA-6490:

Thanks for you response [~mjsax]. I think we need a similar feature in connector because now
if we get a malformed JSON message the connector will fail and will not process any additional
message that are coming after this one. I can work on KIP for solving this issue if the forks
you tagged in agree with what I am saying. Can you also please add me to the contributors
list ?

> JSON SerializationException Stops Connect
> -----------------------------------------
>                 Key: KAFKA-6490
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6490
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 1.0.0
>            Reporter: William R. Speirs
>            Priority: Major
>         Attachments: KAFKA-6490_v1.patch
> If you configure KafkaConnect to parse JSON messages, and you send it a non-JSON message,
the SerializationException message will bubble up to the top, and stop KafkaConnect. While
I understand sending non-JSON to a JSON serializer is a bad idea, I think that a single malformed
message stopping all of KafkaConnect is even worse.
> The data exception is thrown here: [https://github.com/apache/kafka/blob/trunk/connect/json/src/main/java/org/apache/kafka/connect/json/JsonConverter.java#L305]
> From the call here: [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L476]
> This bubbles all the way up to the top, and KafkaConnect simply stops with the message:
{{ERROR WorkerSinkTask\{id=elasticsearch-sink-0} Task threw an uncaught and unrecoverable
exception (org.apache.kafka.connect.runtime.WorkerTask:172)}}
> Thoughts on adding a {{try/catch}} around the {{for}} loop in WorkerSinkTask's {{convertMessages}}
so messages that don't properly parse are logged, but simply ignored? This way KafkaConnect
can keep working even when it encounters a message it cannot decode?

This message was sent by Atlassian JIRA

View raw message