hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hc busy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1016) Reading in map data seems broken
Date Thu, 29 Oct 2009 17:19:59 GMT

    [ https://issues.apache.org/jira/browse/PIG-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771485#action_12771485
] 

hc busy commented on PIG-1016:
------------------------------

Yeah, I ran into that problem immediately. Had to reformat the data  to be

{{\['key'#983579482375984237957294L\]}}

(append 'L' at the end) The alternative...... is something funky like python's auto-promotion.
If it overflows, automatically promote int to long, and then long to BigNumber...

And in the second case is a very very good example. So this is where that 'new Exception("Bad
Compare");' will execute.

For my purpose, I  make sure the map has auto-detectable type and everything works fine.

If your map is known to have String value types, but some are numbers, you could cast it later.

result = foreach input generate (string)map#key

and compare that way.

Some possibilities:
Obviously if we can specify a schema inside the map 

A = load 'data' as (m:map['string_key':string, 'map_key':map[]];

where specified key types are read in as specified type, and then the rest are auto-detect.


Another thing, I guess I'm willing to use a special case of map... if say we made a auto-detecting
map and a byte-array-valued map. then I'm in good shape too.

A = load 'data' as (m:map[]);
B = load 'data' as (m:tTypedMap[]);

But both seems like rather large efforts... (?)

> Reading in map data seems broken
> --------------------------------
>
>                 Key: PIG-1016
>                 URL: https://issues.apache.org/jira/browse/PIG-1016
>             Project: Pig
>          Issue Type: Improvement
>          Components: data
>    Affects Versions: 0.4.0
>            Reporter: hc busy
>             Fix For: 0.5.0
>
>         Attachments: PIG-1016.patch
>
>
> Hi, I'm trying to load a map that has a tuple for value. The read fails in 0.4.0 because
of a misconfiguration in the parser. Where as in almost all documentation it is stated that
value of the map can be any time.
> I've attached a patch that allows us to read in complex objects as value as documented.
I've done simple verification of loading in maps with tuple/map values and writing them back
out using LOAD and STORE. All seems to work fine.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message