hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8328) Duplicate FileSystem Statistics object for 'file' scheme
Date Fri, 27 Apr 2012 21:36:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263993#comment-13263993
] 

Tom White commented on HADOOP-8328:
-----------------------------------

Task's updateCounters() method uses FileSystem.getAllStatistics() to get all the statistics
for all filesystems. However, when there are two Statistics objects for the 'file' scheme,
it will use the first one, whichever that happens to be (it is unstable). MapReduce uses the
RawLocalFileSystem to read and write local files (e.g. when merging), so if the Statistics
object for LocalFileSystem happens to come first out of the Map, then the counters will be
too low.

The quick fix would be to change LocalFileSystem's initialize method to create a single Statistics
object - for RawLocalFileSystem - to bring the behaviour in line with branch-1 (i.e. pre-HADOOP-8013).
There might be a better longer-term fix though.
                
> Duplicate FileSystem Statistics object for 'file' scheme
> --------------------------------------------------------
>
>                 Key: HADOOP-8328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8328
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>
> Because of a change in HADOOP-8013, there are duplicate Statistics objects in FileSystem's
statistics table: one for LocalFileSystem and one for RawLocalFileSystem. This causes MapReduce
local file system counters to be incorrect some of the time. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message