flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicolas Laduguie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-2580) Sink side interceptors
Date Fri, 12 Jul 2019 09:41:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883682#comment-16883682

Nicolas Laduguie commented on FLUME-2580:

Hi, what about this feature request ?

We would appreciate it because we have the use case where we need to consume events from Kafka
and directly write them to HDFS, without passing through a channel which would either not
prevent data loss (file channel, memory channel), or duplicate data to guarantee data integrity
(jdbc channel, kafka channel).

Our last need is to compute HDFS path according to Kafka messages specific fields like date
or anything else, but we can't do it because we need an interceptor either on the channel
side or on the sink side.

Thank you for your support.


> Sink side interceptors
> ----------------------
>                 Key: FLUME-2580
>                 URL: https://issues.apache.org/jira/browse/FLUME-2580
>             Project: Flume
>          Issue Type: New Feature
>            Reporter: Hari Shreedharan
>            Assignee: Hari Shreedharan
>            Priority: Major
>         Attachments: SinkSideInterceptors.pdf
> Currently, we only have source-side interceptors that help routing. But if we use something
like Kafka Channel, having a sink-side interceptor can help us modify events as they come
in. We could also do validation on event schemas and drop them before they hit the sink, rather
than have an infinite loop due to such events. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org

View raw message