beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-881) Provide a PTransform in IOs providing a "standard" Avro IndexedRecord
Date Wed, 02 Nov 2016 16:36:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629505#comment-15629505
] 

Daniel Halperin commented on BEAM-881:
--------------------------------------

I can buy the general use case. But we need the value to be evident before we make an invasive
change across all of Beam.

So to prove this is useful we need a really useful tuple type and demonstrate value to the
rest of the project to support it.

> Provide a PTransform in IOs providing a "standard" Avro IndexedRecord
> ---------------------------------------------------------------------
>
>                 Key: BEAM-881
>                 URL: https://issues.apache.org/jira/browse/BEAM-881
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-extensions
>            Reporter: Jean-Baptiste Onofré
>            Assignee: Jean-Baptiste Onofré
>
> Now, each IO is using a different data format. For instance, the {{JmsIO.Read}} provides
a {{PCollection}} of {{JmsRecord}} (and {{JmsIO.Write}} expects also a {{JmsRecord}}), {{KafkaIO.Read}}
provides a {{PCollection}} of {{KafkaRecord}}.
> It could appear a bit "complex" for users to manipulate such kind of data format: some
users may expect kind of standard format.
> Without modifying the existing IO, we could add a {{PTransform}} (as part of the IO)
that an user can optionally use. This transform will convert the IO data format (let say {{JmsRecord}}
for instance) to a standard Avro {{IndexedRecord}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message