Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Date: Sun, 6 Jul 2014 19:29:33 +0000 (UTC)
From: "Gabriel Reid (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12725501.1404487787378.9481.1404674973811@arcas>
In-Reply-To: <JIRA.12725501.1404487787378@arcas>
References: <JIRA.12725501.1404487787378@arcas>
Subject: [jira] [Commented] (CRUNCH-433) Add support for reading
 specific/reflect data from an Avro MR file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CRUNCH-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053202#comment-14053202 ] 

Gabriel Reid commented on CRUNCH-433:
-------------------------------------

{quote}+1 for the corrected patch, with one request: that BaseAvroTableType be package-scoped instead of public if at all possible.{quote}
Sounds like a good plan. The reason it's public is to use it specifically in AvroTableFileSource, but I think it's easy enough to get around that.

{quote}do we need to add a classifier line to the avro-mapred dependencies in the POM for this stuff to work properly on MR1 vs. MR2?{quote}
I don't think so, but I'm not sure I'm totally following what you mean. The only new thing being done here from avro-mapred is making use of the org.apache.avro.hadoop.io.AvroKeyValue class (basically only for schema creation), so I don't think there's anything that would change there in terms of needing classifiers (or am I missing something?)

> Add support for reading specific/reflect data from an Avro MR file
> ------------------------------------------------------------------
>
>                 Key: CRUNCH-433
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-433
>             Project: Crunch
>          Issue Type: New Feature
>            Reporter: Gabriel Reid
>            Assignee: Gabriel Reid
>         Attachments: CRUNCH-433.patch
>
>
> An Avro Key/Value file written via raw MapReduce contains records that follow the schema generated by the org.apache.avro.hadoop.io.AvroKeyValue class. 
> If these files contain specific or reflection-based records, there is currently no easy way to read them in as specific or reflection records. Using the basic public Crunch APIs, they can only be read as generic records (that also contain generic records).
> A method should be added to the Avros class which allows specifying specific PTypes to be used for reading the underlying data types within a raw MR output file.
> Link to related discussion that inspired this ticket on the user list: http://s.apache.org/es


--
This message was sent by Atlassian JIRA
(v6.2#6252)