hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (MAPREDUCE-6486) File{Input|Output}Format counters not updated when using viewfs
Date Mon, 12 Dec 2016 09:49:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741494#comment-15741494
] 

Vinayakumar B edited comment on MAPREDUCE-6486 at 12/12/16 9:49 AM:
--------------------------------------------------------------------

Since ViewFS is just a wrapper over real filesystems, it would be better just ignore these
stats and dont show it in final counters list.
{{FileSystem.getAllStatistics()}} can filter stats for all filesystems with child filesystems,
or just viewfs.

What do you think [~cnauroth]?


was (Author: vinayrpet):
Since ViewFS is just a wrapper over real filesystems, it would be better just ignore these
stats and dont show it in final counters list.
Since these counters are already updated for resolved filesystems.
{{FileSystem.getAllStatistics()}} can filter stats for all filesystems with child filesystems,
or just viewfs.

What do you think [~cnauroth]?

> File{Input|Output}Format counters not updated when using viewfs
> ---------------------------------------------------------------
>
>                 Key: MAPREDUCE-6486
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6486
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2, task
>            Reporter: Laurent Goujon
>
> MapReduce tasks rely on filesystem stats to update FileInputFormatCounter and FileOutputFormatCounter
bytes counters.
> But if the filesystem is viewfs, stats are not updated and counters are zero. However
the resolved filesystem for the input and output paths have their stats updated, and should
be used by the framework (which could be done by resolving the path first).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message