apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saumya Mohan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (APEXMALHAR-2034) Avro File To POJO Module
Date Tue, 15 Aug 2017 20:57:00 GMT

     [ https://issues.apache.org/jira/browse/APEXMALHAR-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Saumya Mohan updated APEXMALHAR-2034:
-------------------------------------
    Description: 
Issue:
Avro objects are not serialized by Kryo causing the Avro GenericRecord to not be available
to downstream operators if users don't explicitly mark the stream locality at container_local
or thread_local. 

Solution:
This JIRA is used to create a Module on top of AvroFileInputOperator and AvroToPojo operators
such that downstream operators will access POJO instead of Avro GenericRecord.

In this Module, the stream between the two encapsulated operators (AvroFileInputOperator and
AvroToPojo) is set to CONTAINER_LOCAL. 

This new module removes the exposure of GenericRecord to downstream operators and instead
exposes the created POJO to downstream operators.




  was:

This JIRA is used to create a Module on top of AvroFileInputOperator and AvroToPojo operator.
The stream between the two operators will be set to CONTAINER_LOCAL which is required as Avro
objects are not serialized by Kryo. This will help users to directly use the module which
has the locality set to CONTAINER_LOCAL.


> Avro File To POJO Module
> ------------------------
>
>                 Key: APEXMALHAR-2034
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2034
>             Project: Apache Apex Malhar
>          Issue Type: New Feature
>            Reporter: devendra tagare
>            Assignee: Saumya Mohan
>
> Issue:
> Avro objects are not serialized by Kryo causing the Avro GenericRecord to not be available
to downstream operators if users don't explicitly mark the stream locality at container_local
or thread_local. 
> Solution:
> This JIRA is used to create a Module on top of AvroFileInputOperator and AvroToPojo operators
such that downstream operators will access POJO instead of Avro GenericRecord.
> In this Module, the stream between the two encapsulated operators (AvroFileInputOperator
and AvroToPojo) is set to CONTAINER_LOCAL. 
> This new module removes the exposure of GenericRecord to downstream operators and instead
exposes the created POJO to downstream operators.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message