crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CRUNCH-433) Add support for reading specific/reflect data from an Avro MR file
Date Mon, 07 Jul 2014 02:27:33 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Wills resolved CRUNCH-433.
-------------------------------

       Resolution: Fixed
    Fix Version/s: 0.11.0
                   0.8.4

Turns out that the classifier fix in CRUNCH-439 fixes the test failure, so I pushed both to
master and 0.8.

> Add support for reading specific/reflect data from an Avro MR file
> ------------------------------------------------------------------
>
>                 Key: CRUNCH-433
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-433
>             Project: Crunch
>          Issue Type: New Feature
>            Reporter: Gabriel Reid
>            Assignee: Gabriel Reid
>             Fix For: 0.8.4, 0.11.0
>
>         Attachments: CRUNCH-433.1.patch, CRUNCH-433.patch
>
>
> An Avro Key/Value file written via raw MapReduce contains records that follow the schema
generated by the org.apache.avro.hadoop.io.AvroKeyValue class. 
> If these files contain specific or reflection-based records, there is currently no easy
way to read them in as specific or reflection records. Using the basic public Crunch APIs,
they can only be read as generic records (that also contain generic records).
> A method should be added to the Avros class which allows specifying specific PTypes to
be used for reading the underlying data types within a raw MR output file.
> Link to related discussion that inspired this ticket on the user list: http://s.apache.org/es



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message