beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2221) Make KafkaIO coder specification less awkward
Date Tue, 09 May 2017 19:50:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003405#comment-16003405
] 

ASF GitHub Bot commented on BEAM-2221:
--------------------------------------

GitHub user jkff opened a pull request:

    https://github.com/apache/beam/pull/3000

    [BEAM-2221] Cherrypick #2986 into release-2.0.0

    Cherrypick of #2986 KafkaIO API clean up.
    
    R: @davorbonaci 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jkff/incubator-beam cp-2986

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3000.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3000
    
----
commit 7c93ea0ebbc0a36cd280f69984995651eab97d1b
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-05-09T19:42:44Z

    This closes #2986

----


> Make KafkaIO coder specification less awkward
> ---------------------------------------------
>
>                 Key: BEAM-2221
>                 URL: https://issues.apache.org/jira/browse/BEAM-2221
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-extensions
>            Reporter: Eugene Kirpichov
>            Assignee: Raghu Angadi
>             Fix For: 2.0.0
>
>
> readWithCoders and writeWithCoders functions are awkward because they don't emphasize
enough that coders are a poor choice for interpreting wire format.
> The only reason to specify coders in KafkaIO is when coder inference from Deserializer
fails. To emphasize that, let's change the API to be withKeyDeserializer(Deserializer) as
the default choice and withKeyDeserializerAndCoder(Deserializer,Coder) if inference fails;
likewise for value.
> Remove functions using coders to interpret wire format from the API. A common case of
that is Avro and Proto - for that, introduce special helper functions, I guess like withAvro/ProtoKey/Value(...),
which under the hood may be allowed to reuse Avro/ProtoCoder as a utility, but do not expose
this fact.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message