kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6490) JSON SerializationException Stops Connect
Date Wed, 07 Feb 2018 04:50:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354959#comment-16354959

Ewen Cheslack-Postava commented on KAFKA-6490:

[~prasanna1433] I've given you wiki permissions, you should be able to create a page now.

The dead letter queue is something I've specifically heard from a number of users, so it's
definitely in demand. The list I gave is based on a ton of real user feedback, so I feel pretty
confident that it is both a) covering important use cases and b) comprehensive enough to address
the vast majority of use cases. But I'm of course open to discussion of the options. I suspect
*more* options would be the result rather than removing some. If you want to take on this
improvement, we can of course discuss further in the KIP thread :)


With respect to the version, new features should almost universally be worked on in trunk
– older release branches are reserved for bug fixes. In this case, since we just cut 1.1
branches, this would be a candidate for 1.2 / 2.0 and can simply be developed against trunk.

> JSON SerializationException Stops Connect
> -----------------------------------------
>                 Key: KAFKA-6490
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6490
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 1.0.0
>            Reporter: William R. Speirs
>            Priority: Major
>         Attachments: KAFKA-6490_v1.patch
> If you configure KafkaConnect to parse JSON messages, and you send it a non-JSON message,
the SerializationException message will bubble up to the top, and stop KafkaConnect. While
I understand sending non-JSON to a JSON serializer is a bad idea, I think that a single malformed
message stopping all of KafkaConnect is even worse.
> The data exception is thrown here: [https://github.com/apache/kafka/blob/trunk/connect/json/src/main/java/org/apache/kafka/connect/json/JsonConverter.java#L305]
> From the call here: [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L476]
> This bubbles all the way up to the top, and KafkaConnect simply stops with the message:
{{ERROR WorkerSinkTask\{id=elasticsearch-sink-0} Task threw an uncaught and unrecoverable
exception (org.apache.kafka.connect.runtime.WorkerTask:172)}}
> Thoughts on adding a {{try/catch}} around the {{for}} loop in WorkerSinkTask's {{convertMessages}}
so messages that don't properly parse are logged, but simply ignored? This way KafkaConnect
can keep working even when it encounters a message it cannot decode?

This message was sent by Atlassian JIRA

View raw message