hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] Closed: (PIG-1243) Passing Complex map types to and from streaming causes a problem
Date Fri, 14 May 2010 06:47:59 GMT

     [ https://issues.apache.org/jira/browse/PIG-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Dai closed PIG-1243.
---------------------------


> Passing Complex map types to and from streaming causes a problem
> ----------------------------------------------------------------
>
>                 Key: PIG-1243
>                 URL: https://issues.apache.org/jira/browse/PIG-1243
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Viraj Bhat
>            Assignee: Richard Ding
>             Fix For: 0.7.0
>
>
> I have a program which generates different types of Maps fields and stores it into PigStorage.
> {code}
> A = load '/user/viraj/three.txt' using PigStorage();
> B = foreach A generate ['a'#'12'] as b:map[], ['b'#['c'#'12']] as c, ['c'#{(['d'#'15']),(['e'#'16'])}]
as d;
> store B into '/user/viraj/pigtest' using PigStorage();
> {code}
> Now I test the previous output in the below script to make sure I have the right results.
I also pass this data to a Perl script and I observe that the complex Map types I have generated,
are lost when I get the result back.
> {code}
> DEFINE CMD `simple.pl` SHIP('simple.pl');
> A = load '/user/viraj/pigtest' using PigStorage() as (simpleFields, mapFields, mapListFields);
> B = foreach A generate $0, $1, $2;
> dump B;
> C = foreach A generate  (chararray)simpleFields#'a' as value, $0,$1,$2;
> D = stream C through CMD as (a0:map[], a1:map[], a2:map[]);
> dump D;
> {code}
> dumping B results in:
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> dumping D results in:
> ([a#12],,)
> ([a#12],,)
> ([a#12],,)
> The Perl script used here is:
> {code}
> #!/usr/local/bin/perl
> use warnings;
> use strict;
> while(<>) {
>     my($bc,$s,$m,$l)=split/\t/;
>     print("$s\t$m\t$l");
> }
> {code}
> Is there an issue with handling of complex Map fields within streaming? How can I fix
this to obtain the right result?
> Viraj

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message