pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Viraj Bhat (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-3322) AVRO: AvroStorage give NPE on reading file with union as top level schema
Date Tue, 04 Jun 2013 00:20:21 GMT

     [ https://issues.apache.org/jira/browse/PIG-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Viraj Bhat updated PIG-3322:
----------------------------

    Attachment:     (was: PIG-3322_2.patch)
    
> AVRO: AvroStorage give NPE on reading file with union as top level schema
> -------------------------------------------------------------------------
>
>                 Key: PIG-3322
>                 URL: https://issues.apache.org/jira/browse/PIG-3322
>             Project: Pig
>          Issue Type: Bug
>          Components: piggybank
>    Affects Versions: 0.11.2
>            Reporter: Egil Sorensen
>            Assignee: Viraj Bhat
>              Labels: patch
>             Fix For: 0.12
>
>         Attachments: PIG-3322_3.patch, test_loadavrowithnulls.avro
>
>
> I am getting NPE when loading a file with AvroStorage a file that has schema like:
> {code}
> ["null",{"type":"record","name":"TUPLE_0","fields":[{"name":"name","type":["null","string"],"doc":"autogenerated
from Pig Field Schema"},{"name":"age","type":["null","int"],"doc":"autogenerated from Pig
Field Schema"},{"name":"gpa","type":["null","double"],"doc":"autogenerated from Pig Field
Schema"}]}]
> {code}
> E.g. see the e2e style test, which fails on this:
> {code}
>                         {
>                         'num' => 4,
>                         # storing file with Pig type tuple relying on conversion to record
>                         # loading using stored schemas 
>                         'notmq' => 1,
>                         'pig' => q\
> a = load ':INPATH:/singlefile/studentcomplextab10k' using PigStorage() as (m:[], t:(name:chararray,
age:int, gpa:double), b:{t:(name:chararray, age:int, gpa:double)});
> b = foreach a generate t;
> describe b;
> store b into ':OUTPATH:.intermediate' USING org.apache.pig.piggybank.storage.avro.AvroStorage();
> exec;
> -- Read back what was stored with Avro
> u = load ':OUTPATH:.intermediate' USING org.apache.pig.piggybank.storage.avro.AvroStorage();
> describe u;
> store u into ':OUTPATH:';
> \,
>                         'verify_pig_script' => q\
> a = load ':INPATH:/singlefile/studentcomplextab10k' using PigStorage() as (m:[], t:(name:chararray,
age:int, gpa:double), b:{t:(name:chararray, age:int, gpa:double)});
> b = foreach a generate t;
> describe b;
> store b into ':OUTPATH:';
> \,
>                         },
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message