Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Wed, 8 Mar 2017 05:27:38 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.12954416.1459283812000.8852.1488950858107@Atlassian.JIRA>
In-Reply-To: <JIRA.12954416.1459283812000@Atlassian.JIRA>
References: <JIRA.12954416.1459283812000@Atlassian.JIRA> <JIRA.12954416.1459283812867@jira-lw-us.apache.org>
Subject: [jira] [Commented] (FLINK-3679) Allow Kafka consumer to skip
 corrupted messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 08 Mar 2017 05:27:44 -0000


    [ https://issues.apache.org/jira/browse/FLINK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900735#comment-15900735 ] 

ASF GitHub Bot commented on FLINK-3679:
---------------------------------------

Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/3314
  
    LGTM, I'll proceed to merge this later today.
    One minor problem: the offset state still isn't updated if `record == null`. We need to do the checking in the synchronize block in the `emitRecord*` methods.
    
    It's a simple fix, so I can do it while merging :-)


> Allow Kafka consumer to skip corrupted messages
> -----------------------------------------------
>
>                 Key: FLINK-3679
>                 URL: https://issues.apache.org/jira/browse/FLINK-3679
>             Project: Flink
>          Issue Type: Bug
>          Components: DataStream API, Kafka Connector
>            Reporter: Jamie Grier
>            Assignee: Haohui Mai
>
> There are a couple of issues with the DeserializationSchema API that I think should be improved.  This request has come to me via an existing Flink user.
> The main issue is simply that the API assumes that there is a one-to-one mapping between input and outputs.  In reality there are scenarios where one input message (say from Kafka) might actually map to zero or more logical elements in the pipeline.
> Particularly important here is the case where you receive a message from a source (such as Kafka) and say the raw bytes don't deserialize properly.  Right now the only recourse is to throw IOException and therefore fail the job.  
> This is definitely not good since bad data is a reality and failing the job is not the right option.  If the job fails we'll just end up replaying the bad data and the whole thing will start again.
> Instead in this case it would be best if the user could just return the empty set.
> The other case is where one input message should logically be multiple output messages.  This case is probably less important since there are other ways to do this but in general it might be good to make the DeserializationSchema.deserialize() method return a collection rather than a single element.
> Maybe we need to support a DeserializationSchema variant that has semantics more like that of FlatMap.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)