hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Blue (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6639) Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
Date Fri, 19 Feb 2016 18:05:18 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ryan Blue updated MAPREDUCE-6639:
---------------------------------
    Attachment: MAPREDUCE-6639.1.patch

Attaching a fix. This is slightly different from what I suggested above. Because a result
needs to be returned for the future, I've added an unknownError field to the results that
the futures check. If there is an error, the futures call registerError.

> Process hangs in LocatedFileStatusFetcher if FileSystem.get throws
> ------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6639
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6639
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 2.7.2
>            Reporter: Ryan Blue
>            Assignee: Ryan Blue
>         Attachments: MAPREDUCE-6639.1.patch
>
>
> ListLocatedFileStatusFetcher uses a thread pool, but one of the Callable thread functions,
[{{ProcessInitialInputPathCallable}}|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java#L306],
doesn't catch exceptions (the others do). When an exception is thrown, the thread exists and
doesn't signal the error to the calling thread, which continues waiting to be signaled. This
can happen when a FS implementation cannot be found.
> The solution is to use a try-catch around the implementation of {{ProcessInitialInputPathCallable#call}}
that calls {{registerError}} when any exception is caught.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message