hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1296) Skewed join fail due to negative partition index
Date Mon, 15 Mar 2010 17:31:27 GMT

    [ https://issues.apache.org/jira/browse/PIG-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845418#action_12845418
] 

Daniel Dai commented on PIG-1296:
---------------------------------

In SkewedPartitioner.java, we have:

return (Math.abs(keyTuple.hashCode()) % totalReducers);

In javadoc, we have the note:
Math.abs(int): if the argument is equal to the value of Integer.MIN_VALUE, the most negative
representable int value, the result is that same value, which is negative.

We hit exactly the same situation in this case.

> Skewed join fail due to negative partition index
> ------------------------------------------------
>
>                 Key: PIG-1296
>                 URL: https://issues.apache.org/jira/browse/PIG-1296
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.7.0
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.7.0
>
>         Attachments: PIG-1296-1.patch
>
>
> Skewed join throw stack:
> java.io.IOException: Illegal partition for Partition: -1 Null: false index: 0 (fc52di95l6m3j,20100210)
(-3648)
>             at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:904)
>             at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:541)
>             at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$MapWithPartitionIndex.collect(PigMapReduce.java:187)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$MapWithPartitionIndex.runPipeline(PigMapReduce.java:206)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:227)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:52)
>             at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>             at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>             at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>             at org.apache.hadoop.mapred.Child.main(Child.java:159)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message