From commits-return-3639-apmail-pig-commits-archive=pig.apache.org@pig.apache.org Thu Dec 09 18:39:15 2010 Return-Path: Delivered-To: apmail-pig-commits-archive@www.apache.org Received: (qmail 63772 invoked from network); 9 Dec 2010 18:39:15 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 9 Dec 2010 18:39:15 -0000 Received: (qmail 63176 invoked by uid 500); 9 Dec 2010 18:39:15 -0000 Delivered-To: apmail-pig-commits-archive@pig.apache.org Received: (qmail 63159 invoked by uid 500); 9 Dec 2010 18:39:15 -0000 Mailing-List: contact commits-help@pig.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@pig.apache.org Delivered-To: mailing list commits@pig.apache.org Received: (qmail 63149 invoked by uid 99); 9 Dec 2010 18:39:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 18:39:15 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 18:39:12 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 27AE123889E9; Thu, 9 Dec 2010 18:38:51 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1044075 - in /pig/trunk: CHANGES.txt src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java Date: Thu, 09 Dec 2010 18:38:51 -0000 To: commits@pig.apache.org From: yanz@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20101209183851.27AE123889E9@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: yanz Date: Thu Dec 9 18:38:50 2010 New Revision: 1044075 URL: http://svn.apache.org/viewvc?rev=1044075&view=rev Log: PIG-1757: After split combination, the number of maps may vary slightly (yanz) Modified: pig/trunk/CHANGES.txt pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java Modified: pig/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/pig/trunk/CHANGES.txt?rev=1044075&r1=1044074&r2=1044075&view=diff ============================================================================== --- pig/trunk/CHANGES.txt (original) +++ pig/trunk/CHANGES.txt Thu Dec 9 18:38:50 2010 @@ -230,6 +230,8 @@ PIG-1309: Map-side Cogroup (ashutoshc) BUG FIXES +PIG-1757: After split combination, the number of maps may vary slightly (yanz) + PIG-1760: Need to report progress in all databags (rding) PIG-1709: Skewed join use fewer reducer for extreme large key (daijy) Modified: pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java URL: http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java?rev=1044075&r1=1044074&r2=1044075&view=diff ============================================================================== --- pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java (original) +++ pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java Thu Dec 9 18:38:50 2010 @@ -27,6 +27,7 @@ import java.util.List; import java.util.Map; import java.util.Comparator; import java.util.Collections; +import java.util.Arrays; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -375,6 +376,8 @@ public class MapRedUtil { } else { ComparableSplit csplit = new ComparableSplit(split, comparableSplitId++); String[] locations = split.getLocations(); + // sort the locations to stabilize the number of maps: PIG-1757 + Arrays.sort(locations); HashSet locationSeen = new HashSet(); for (String location : locations) {