Mailing-List: contact dev-help@pig.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@pig.apache.org
Date: Wed, 3 Dec 2014 09:45:12 +0000 (UTC)
From: "Michael Prim (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Message-ID: <JIRA.12754760.1415801232000.66854.1417599912459@Atlassian.JIRA>
In-Reply-To: <JIRA.12754760.1415801232000@Atlassian.JIRA>
References: <JIRA.12754760.1415801232000@Atlassian.JIRA>
 <JIRA.12754760.1415801232302@arcas>
Subject: [jira] [Commented] (PIG-4326) AvroStorageSchemaConversionUtilities
 does not properly convert schema for maps of arrays of records
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/PIG-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232823#comment-14232823 ] 

Michael Prim commented on PIG-4326:
-----------------------------------

[~daijy] Did you have a chance to test the new patch? Would like to get this done and out of my scope :)

> AvroStorageSchemaConversionUtilities does not properly convert schema for maps of arrays of records
> ---------------------------------------------------------------------------------------------------
>
>                 Key: PIG-4326
>                 URL: https://issues.apache.org/jira/browse/PIG-4326
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.12.0, 0.13.0
>            Reporter: Michael Prim
>            Assignee: Michael Prim
>             Fix For: 0.15.0
>
>         Attachments: PIG-4326-0.patch, mapsOfArraysOfRecords.patch, supportForMapsOfArraysOfRecords.patch
>
>
> I tried to convert the avro schema of a map of arrays of records into the proper pig schema and got always empty map schemas in pig.
> The reason is that the AvroStorageSchemaConversionUtilities does only assume records or primitive types as content of the map. However, a map of arrays, or a map of map, could have a schema itself and requires recursive calling to derive the full schema.
> I wrote a unit test to test for maps of arrays of records which fails with every pig release since the AvroStorage was rewritten (I think this was in 0.12), and there have been no changes since then in the trunk. 
> Further the attached patch contains the (rather simple) fix that makes the schema conversion utils succeed.
> Would appreciate further comments and if this can be included upstream.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)