From issues-return-908-archive-asf-public=cust-asf.ponee.io@flume.apache.org Fri Jul 12 09:41:01 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id A8DFC180675 for ; Fri, 12 Jul 2019 11:41:01 +0200 (CEST) Received: (qmail 20556 invoked by uid 500); 12 Jul 2019 09:41:01 -0000 Mailing-List: contact issues-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flume.apache.org Delivered-To: mailing list issues@flume.apache.org Received: (qmail 20539 invoked by uid 99); 12 Jul 2019 09:41:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Jul 2019 09:41:01 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 7080AE2A06 for ; Fri, 12 Jul 2019 09:41:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2F23B26581 for ; Fri, 12 Jul 2019 09:41:00 +0000 (UTC) Date: Fri, 12 Jul 2019 09:41:00 +0000 (UTC) From: "Nicolas Laduguie (JIRA)" To: issues@flume.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (FLUME-2580) Sink side interceptors MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/FLUME-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883682#comment-16883682 ] Nicolas Laduguie commented on FLUME-2580: ----------------------------------------- Hi, what about this feature request ? We would appreciate it because we have the use case where we need to consume events from Kafka and directly write them to HDFS, without passing through a channel which would either not prevent data loss (file channel, memory channel), or duplicate data to guarantee data integrity (jdbc channel, kafka channel). Our last need is to compute HDFS path according to Kafka messages specific fields like date or anything else, but we can't do it because we need an interceptor either on the channel side or on the sink side. Thank you for your support. Regards > Sink side interceptors > ---------------------- > > Key: FLUME-2580 > URL: https://issues.apache.org/jira/browse/FLUME-2580 > Project: Flume > Issue Type: New Feature > Reporter: Hari Shreedharan > Assignee: Hari Shreedharan > Priority: Major > Attachments: SinkSideInterceptors.pdf > > > Currently, we only have source-side interceptors that help routing. But if we use something like Kafka Channel, having a sink-side interceptor can help us modify events as they come in. We could also do validation on event schemas and drop them before they hit the sink, rather than have an infinite loop due to such events. -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org For additional commands, e-mail: issues-help@flume.apache.org