hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hc busy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1016) Reading in map data seems broken
Date Thu, 29 Oct 2009 17:19:59 GMT

    [ https://issues.apache.org/jira/browse/PIG-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771485#action_12771485

hc busy commented on PIG-1016:

Yeah, I ran into that problem immediately. Had to reformat the data  to be


(append 'L' at the end) The alternative...... is something funky like python's auto-promotion.
If it overflows, automatically promote int to long, and then long to BigNumber...

And in the second case is a very very good example. So this is where that 'new Exception("Bad
Compare");' will execute.

For my purpose, I  make sure the map has auto-detectable type and everything works fine.

If your map is known to have String value types, but some are numbers, you could cast it later.

result = foreach input generate (string)map#key

and compare that way.

Some possibilities:
Obviously if we can specify a schema inside the map 

A = load 'data' as (m:map['string_key':string, 'map_key':map[]];

where specified key types are read in as specified type, and then the rest are auto-detect.

Another thing, I guess I'm willing to use a special case of map... if say we made a auto-detecting
map and a byte-array-valued map. then I'm in good shape too.

A = load 'data' as (m:map[]);
B = load 'data' as (m:tTypedMap[]);

But both seems like rather large efforts... (?)

> Reading in map data seems broken
> --------------------------------
>                 Key: PIG-1016
>                 URL: https://issues.apache.org/jira/browse/PIG-1016
>             Project: Pig
>          Issue Type: Improvement
>          Components: data
>    Affects Versions: 0.4.0
>            Reporter: hc busy
>             Fix For: 0.5.0
>         Attachments: PIG-1016.patch
> Hi, I'm trying to load a map that has a tuple for value. The read fails in 0.4.0 because
of a misconfiguration in the parser. Where as in almost all documentation it is stated that
value of the map can be any time.
> I've attached a patch that allows us to read in complex objects as value as documented.
I've done simple verification of loading in maps with tuple/map values and writing them back
out using LOAD and STORE. All seems to work fine.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message