hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1016) Reading in map data seems broken
Date Wed, 24 Feb 2010 02:01:28 GMT

    [ https://issues.apache.org/jira/browse/PIG-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837574#action_12837574
] 

Daniel Dai commented on PIG-1016:
---------------------------------

Hi, busy,
I checked your code, seems your patch assume PIG-1016.patch checked in. If I understand correctly,
there are inconsistency in this approach. In your code, you allow map value to be any type.
However, internally Pig always assume map value to be bytearray. So Pig will choose to use
PigBytesRawComparator. And you further modify PigBytesRawComparator to handle all data type.
This logic is very confusing. Further, TextDataParser itself if bogus since it will guess
the data type based on the content. 

In PIG-613, we reiterate that map value is bytearray. However, we fixed the code which can
cast bytearray to map/tuple/bag correctly. I verified the test case you gave, and it works.

{code}
A= load '9.txt' as (data:map[]);
B= foreach A generate (int)(data#'a'), (chararray)(data#'b'),(tuple(map[]))(data#'c');
C= order B by $0;
dump C;
{code}
Result:
(1,'a',(1,2,3))
(2,'d',(1,2,3))
(3,'c',(1,2,3))

{code}
D= order B by $1;
dump D;
{code}
Result:
(1,'a',(1,2,3))
(3,'c',(1,2,3))
(2,'d',(1,2,3))

{code}
describe B;
{code}
Result:
B: {int,chararray,(map[ ])}

Do you have other use cases which PIG-613 cannot address?

> Reading in map data seems broken
> --------------------------------
>
>                 Key: PIG-1016
>                 URL: https://issues.apache.org/jira/browse/PIG-1016
>             Project: Pig
>          Issue Type: Improvement
>          Components: data
>    Affects Versions: 0.4.0
>            Reporter: hc busy
>         Attachments: PIG-1016.patch
>
>
> Hi, I'm trying to load a map that has a tuple for value. The read fails in 0.4.0 because
of a misconfiguration in the parser. Where as in almost all documentation it is stated that
value of the map can be any time.
> I've attached a patch that allows us to read in complex objects as value as documented.
I've done simple verification of loading in maps with tuple/map values and writing them back
out using LOAD and STORE. All seems to work fine.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message