Return-Path: X-Original-To: apmail-pig-dev-archive@www.apache.org Delivered-To: apmail-pig-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 134BBC65C for ; Fri, 14 Nov 2014 10:59:34 +0000 (UTC) Received: (qmail 18998 invoked by uid 500); 14 Nov 2014 10:59:33 -0000 Delivered-To: apmail-pig-dev-archive@pig.apache.org Received: (qmail 18938 invoked by uid 500); 14 Nov 2014 10:59:33 -0000 Mailing-List: contact dev-help@pig.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@pig.apache.org Delivered-To: mailing list dev@pig.apache.org Received: (qmail 18922 invoked by uid 500); 14 Nov 2014 10:59:33 -0000 Delivered-To: apmail-hadoop-pig-dev@hadoop.apache.org Received: (qmail 18919 invoked by uid 99); 14 Nov 2014 10:59:33 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Nov 2014 10:59:33 +0000 Date: Fri, 14 Nov 2014 10:59:33 +0000 (UTC) From: "Michael Prim (JIRA)" To: pig-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PIG-4326) AvroStorageSchemaConversionUtilities does not properly convert schema for maps of arrays of records MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212115#comment-14212115 ] Michael Prim commented on PIG-4326: ----------------------------------- Thanks for the feedback, I realized that during development and was actually also a bit surprised. However removing this extra layer, breaks the already existing testLoadRecordsWithMapOfRecords test, and would be not backward compatible. Further, if you create a map> using avro avdl files, it is just syntactic sugar for actually having some map where WrapperRecord has one field, namely an array of InnerRecord. As neither the WrapperRecord has an alias, nor the array of InnerRecords itself, it is a bit confusing that both get the "array" alias. So we could stick to the old behavior for records and drop the wrapping tuple only for maps and arrays, but then the resulting output will look different than the input I think. > AvroStorageSchemaConversionUtilities does not properly convert schema for maps of arrays of records > --------------------------------------------------------------------------------------------------- > > Key: PIG-4326 > URL: https://issues.apache.org/jira/browse/PIG-4326 > Project: Pig > Issue Type: Bug > Components: impl > Affects Versions: 0.12.0, 0.13.0 > Reporter: Michael Prim > Assignee: Michael Prim > Fix For: 0.15.0 > > Attachments: mapsOfArraysOfRecords.patch > > > I tried to convert the avro schema of a map of arrays of records into the proper pig schema and got always empty map schemas in pig. > The reason is that the AvroStorageSchemaConversionUtilities does only assume records or primitive types as content of the map. However, a map of arrays, or a map of map, could have a schema itself and requires recursive calling to derive the full schema. > I wrote a unit test to test for maps of arrays of records which fails with every pig release since the AvroStorage was rewritten (I think this was in 0.12), and there have been no changes since then in the trunk. > Further the attached patch contains the (rather simple) fix that makes the schema conversion utils succeed. > Would appreciate further comments and if this can be included upstream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)