kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect
Date Mon, 13 Jun 2016 23:50:57 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328647#comment-15328647

Ewen Cheslack-Postava commented on KAFKA-3209:

[~pablocasares] I don't think anybody is actively working on this -- Nisarg had started but
I'm not sure if they have dropped work on it for now since we haven't seen anything for awhile.
First step would be to sketch out a design and put together a KIP. We'll want to make sure
the use cases and how the proposed interfaces work for them is clearly documented. Especially
important will be discussion around configuration -- how will transforms be configured in
the Connector configs, possibly allowing for multiple transformations while keeping configuration
easy? Additionally, if we are going to ship anything with Kafka to provide very commonly needed
transformations (dropping & renaming fields, flattening hierarchical structures, filters),
we'll probably want to include a full description of that set in the KIP as well. (Alternatively,
we might not ship any, but then they obviously aren't as useful out of the box -- like connectors
and converters, you'll need to pull in separate jars for transforms). I imagine the implementation
will be pretty straightforward once we've got it specced out.

> Support single message transforms in Kafka Connect
> --------------------------------------------------
>                 Key: KAFKA-3209
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3209
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Neha Narkhede
> Users should be able to perform light transformations on messages between a connector
and Kafka. This is needed because some transformations must be performed before the data hits
Kafka (e.g. filtering certain types of events or PII filtering). It's also useful for very
light, single-message modifications that are easier to perform inline with the data import/export.

This message was sent by Atlassian JIRA

View raw message