kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4714) Implement remaining KIP-66 SMTs
Date Wed, 20 Sep 2017 17:05:02 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173506#comment-16173506

Ewen Cheslack-Postava commented on KAFKA-4714:

[~dhananjaydp] You cannot do that with SMTs, you would need to define two connectors as you
describe. The KIP probably could have included some more information about why. Your use case
is really a special case of the more general problem of a single record generating multiple
outputs, with some of those outputs possibly transformed in some way. Supporting this would
significantly complicate the system, in ways that it would be difficult to provide the guarantees
we provide today (e.g. at-least once delivery). It messes with some core concepts in Connect,
such as having unique (partition, offset) values for each message. Complex processing topologies
like this are better suited to Kafka Streams, which was built just for that purpose.

Note that you *can* do this without SMTs if you write your own connector that can safely handle
each message resulting in multiple writes to some output system. You could still use the transformations
internally in that connector if you wanted to reuse the existing transformation code, they
just wouldn't be configured via the standard mechanism in Connect.

As for the specific use case of the original message, if someone made a strong case for it
and a KIP describing how it would be implemented, sure. We're open to plenty of ideas about
how to improve the framework.

> Implement remaining KIP-66 SMTs
> -------------------------------
>                 Key: KAFKA-4714
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4714
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>            Reporter: Ewen Cheslack-Postava
>            Assignee: Ewen Cheslack-Postava
>             Fix For:
> Three didn't make it for the release: Flatten, Cast, and TimestampConverter.

This message was sent by Atlassian JIRA

View raw message