Mailing-List: contact dev-help@pig.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@pig.apache.org
Date: Tue, 2 Dec 2014 16:35:13 +0000 (UTC)
From: "Travis Woodruff (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Message-ID: <JIRA.12550580.1334178012000.59393.1417538113689@Atlassian.JIRA>
In-Reply-To: <JIRA.12550580.1334178012000@Atlassian.JIRA>
References: <JIRA.12550580.1334178012000@Atlassian.JIRA>
 <JIRA.12550580.1334178012437@arcas>
Subject: [jira] [Commented] (PIG-2647) Split Combining drops splits with
 empty getLocations()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/PIG-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231705#comment-14231705 ] 

Travis Woodruff commented on PIG-2647:
--------------------------------------

Ran into this in a production environment, We lost several DataNodes, and this caused incorrect results to be produced (!!!), as files with no available blocks were silently ignored.

> Split Combining drops splits with empty getLocations()
> ------------------------------------------------------
>
>                 Key: PIG-2647
>                 URL: https://issues.apache.org/jira/browse/PIG-2647
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Alex Levenson
>            Assignee: Travis Woodruff
>
> in:
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil#getCombinePigSplits
> which is used by PigInputFormat
> There is an assumption that every split's getLocations() will return a non-empty array.
> If the following criteria are met:
> 1) Split combining is turned on
> 2) There is more than one split
> 3) There is at least one split that is smaller than the maxCombineSplitSize
> splits with empty getLocations() will simply be dropped (ignored) without warning.
> The hadoop API does not specify that all splits must return a location and there are cases where a split may want to return no locations (if the data is not in HDFS for example, or if the data is a directory full of HDFS files in which case there's not much gained by having locality)
> This is due to the implementation of org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil#getCombinePigSplits
> scans all splits eligible for combining and creates a map of Nodes -> splits, then laster iterates through the MAP (not the splits) to do the combining.
> One solution would be to inject a dummy "empty node" into the map.
> Overall the logic in getCombinePigSplits is very complicated and has a lot of edge cases, it might be worth cleaning up.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)