pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Travis Woodruff (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-2647) Split Combining drops splits with empty getLocations()
Date Tue, 02 Dec 2014 16:35:13 GMT

    [ https://issues.apache.org/jira/browse/PIG-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231705#comment-14231705

Travis Woodruff commented on PIG-2647:

Ran into this in a production environment, We lost several DataNodes, and this caused incorrect
results to be produced (!!!), as files with no available blocks were silently ignored.

> Split Combining drops splits with empty getLocations()
> ------------------------------------------------------
>                 Key: PIG-2647
>                 URL: https://issues.apache.org/jira/browse/PIG-2647
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Alex Levenson
>            Assignee: Travis Woodruff
> in:
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil#getCombinePigSplits
> which is used by PigInputFormat
> There is an assumption that every split's getLocations() will return a non-empty array.
> If the following criteria are met:
> 1) Split combining is turned on
> 2) There is more than one split
> 3) There is at least one split that is smaller than the maxCombineSplitSize
> splits with empty getLocations() will simply be dropped (ignored) without warning.
> The hadoop API does not specify that all splits must return a location and there are
cases where a split may want to return no locations (if the data is not in HDFS for example,
or if the data is a directory full of HDFS files in which case there's not much gained by
having locality)
> This is due to the implementation of org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil#getCombinePigSplits
> scans all splits eligible for combining and creates a map of Nodes -> splits, then
laster iterates through the MAP (not the splits) to do the combining.
> One solution would be to inject a dummy "empty node" into the map.
> Overall the logic in getCombinePigSplits is very complicated and has a lot of edge cases,
it might be worth cleaning up.

This message was sent by Atlassian JIRA

View raw message