pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5040) Order by and CROSS partitioning is not deterministic due to usage of Random
Date Fri, 14 Oct 2016 03:29:20 GMT

    [ https://issues.apache.org/jira/browse/PIG-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574041#comment-15574041

Rohini Palaniswamy commented on PIG-5040:

bq. Is it possible to reuse PigConstants.TASK_INDEX?
 Why? TASK_ID is also available in jobconf and provides better hashcode being a String than
task index which is just a integer whose hashcode is same as the integer.  

Changes to LocalMapReduceSimulator.java were done for TestExampleGenerator which tests PigServer.getExamples().

> Order by and CROSS partitioning is not deterministic due to usage of Random
> ---------------------------------------------------------------------------
>                 Key: PIG-5040
>                 URL: https://issues.apache.org/jira/browse/PIG-5040
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>            Priority: Critical
>             Fix For: 0.17.0, 0.16.1
>         Attachments: PIG-5040-1-nowhitespacechanges.patch, PIG-5040-1.patch
> Maps can be rerun due to shuffle fetch failures. Half of the reducers can end up successfully
pulling partitions from first run of the map while other half could pull from the rerun after
shuffle fetch failures. If the data is not partitioned by the Partitioner exactly the same
way every time then it could lead to incorrect results (loss of records and duplicated records).
Even though issue has existed for 8 years now with order by and affects mapreduce as well
found this with Tez where the frequency of rerun due to shuffle fetch failures is high (Order
by partitioner gets its data from a 1-1 edge, so there are no retries and shuffle fetch failures
trigger a rerun immediately).

This message was sent by Atlassian JIRA

View raw message