hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6486) File{Input|Output}Format counters not updated when using viewfs
Date Mon, 12 Dec 2016 09:48:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741494#comment-15741494

Vinayakumar B commented on MAPREDUCE-6486:

Since ViewFS is just a wrapper over real filesystems, it would be better just ignore these
stats and dont show it in final counters list.
Since these counters are already updated for resolved filesystems.
{{FileSystem.getAllStatistics()}} can filter stats for all filesystems with child filesystems,
or just viewfs.

What do you think [~cnauroth]?

> File{Input|Output}Format counters not updated when using viewfs
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-6486
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6486
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2, task
>            Reporter: Laurent Goujon
> MapReduce tasks rely on filesystem stats to update FileInputFormatCounter and FileOutputFormatCounter
bytes counters.
> But if the filesystem is viewfs, stats are not updated and counters are zero. However
the resolved filesystem for the input and output paths have their stats updated, and should
be used by the framework (which could be done by resolving the path first).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message