beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré (JIRA) <j...@apache.org>
Subject [jira] [Created] (BEAM-881) Provide a PTransform in IOs providing a "standard" Avro IndexedRecord
Date Wed, 02 Nov 2016 15:41:59 GMT
Jean-Baptiste Onofré created BEAM-881:
-----------------------------------------

             Summary: Provide a PTransform in IOs providing a "standard" Avro IndexedRecord
                 Key: BEAM-881
                 URL: https://issues.apache.org/jira/browse/BEAM-881
             Project: Beam
          Issue Type: New Feature
          Components: sdk-java-extensions
            Reporter: Jean-Baptiste Onofré
            Assignee: Jean-Baptiste Onofré


Now, each IO is using a different data format. For instance, the {{JmsIO.Read}} provides a
{{PCollection}} of {{JmsRecord}} (and {{JmsIO.Write}} expects also a {{JmsRecord}}), {{KafkaIO.Read}}
provides a {{PCollection}} of {{KafkaRecord}}.

It could appear a bit "complex" for users to manipulate such kind of data format: some users
may expect kind of standard format.

Without modifying the existing IO, we could add a {{PTransform}} (as part of the IO) that
an user can optionally use. This transform will convert the IO data format (let say {{JmsRecord}}
for instance) to a standard Avro {{IndexedRecord}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message