flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jamie Grier (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-3679) DeserializationSchema should handle zero or more outputs for every input
Date Tue, 29 Mar 2016 20:37:25 GMT
Jamie Grier created FLINK-3679:

             Summary: DeserializationSchema should handle zero or more outputs for every input
                 Key: FLINK-3679
                 URL: https://issues.apache.org/jira/browse/FLINK-3679
             Project: Flink
          Issue Type: Bug
          Components: DataStream API
            Reporter: Jamie Grier

There are a couple of issues with the DeserializationSchema API that I think should be improved.
 This request has come to me via an existing Flink user.

The main issue is simply that the API assumes that there is a one-to-one mapping between input
and outputs.  In reality there are scenarios where one input message (say from Kafka) might
actually map to zero or more logical elements in the pipeline.

Particularly important here is the case where you receive a message from a source (such as
Kafka) and say the raw bytes don't deserialize properly.  Right now the only recourse is to
throw IOException and therefore fail the job.  

This is definitely not good since bad data is a reality and failing the job is not the right
option.  If the job fails we'll just end up replaying the bad data and the whole thing will
start again.

Instead in this case it would be best if the user could just return the empty set.

The other case is where one input message should logically be multiple output messages.  This
case is probably less important since there are other ways to do this but in general it might
be good to make the DeserializationSchema.deserialize() method return a collection rather
than a single element.

Maybe we need to support a DeserializationSchema variant that has semantics more like that
of FlatMap.

This message was sent by Atlassian JIRA

View raw message