Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Wed, 1 Mar 2017 04:14:45 +0000 (UTC)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.13046979.1488290294000.26203.1488341685610@Atlassian.JIRA>
In-Reply-To: <JIRA.13046979.1488290294000@Atlassian.JIRA>
References: <JIRA.13046979.1488290294000@Atlassian.JIRA> <JIRA.13046979.1488290294718@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-1573) KafkaIO does not allow using Kafka
 serializers and deserializers
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 01 Mar 2017 04:14:49 -0000


    [ https://issues.apache.org/jira/browse/BEAM-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889471#comment-15889471 ] 

Raghu Angadi commented on BEAM-1573:
------------------------------------

[~peay], it is not clear what you are trying to do that is not possible now. Can you paste (psuedo) code for either Beam pipeline (or even simple {{consume(KafkaRecord record)}} method) to illustrate what you want to do? Btw, is this for source or sink?

Note that if you don't do '.withoutMetaData()' on KafkaIO.read(), it return KafkaRecord<K, V>, which does contain topic, partition and other metadata.


> KafkaIO does not allow using Kafka serializers and deserializers
> ----------------------------------------------------------------
>
>                 Key: BEAM-1573
>                 URL: https://issues.apache.org/jira/browse/BEAM-1573
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>    Affects Versions: 0.4.0, 0.5.0
>            Reporter: peay
>            Assignee: Raghu Angadi
>            Priority: Minor
>
> KafkaIO does not allow to override the serializer and deserializer settings of the Kafka consumer and producers it uses internally. Instead, it allows to set a `Coder`, and has a simple Kafka serializer/deserializer wrapper class that calls the coder.
> I appreciate that allowing to use Beam coders is good and consistent with the rest of the system. However, is there a reason to completely disallow to use custom Kafka serializers instead?
> This is a limitation when working with an Avro schema registry for instance, which requires custom serializers. One can write a `Coder` that wraps a custom Kafka serializer, but that means two levels of un-necessary wrapping.
> In addition, the `Coder` abstraction is not equivalent to Kafka's `Serializer` which gets the topic name as input. Using a `Coder` wrapper would require duplicating the output topic setting in the argument to `KafkaIO` and when building the wrapper, which is not elegant and error prone.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)