hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13031) Refactor rack-aware counters from FileSystemStorageStatistics to HFDS specific StorageStatistics
Date Tue, 07 Jun 2016 00:42:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317565#comment-15317565
] 

Ming Ma commented on HADOOP-13031:
----------------------------------

Thanks [~liuml07]. Do you have a MAPREDUCE jira to consume any StorageStatistics and expose
them to MR counters? MAPREDUCE-6660 was created specifically for network distance, but maybe
it is better to solve the general case.

> Refactor rack-aware counters from FileSystemStorageStatistics to HFDS specific StorageStatistics
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-13031
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13031
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>    Affects Versions: 2.9.0
>            Reporter: Mingliang Liu
>
> [HADOOP-13065] added a new interface for retrieving FS and FC Statistics. This jira is
to refactor the code that maintains rack-aware read metrics to use the newly added StorageStatistics.
Specially,
> # Rack-aware read bytes metrics is mostly specific to HDFS. For example, local file system
doesn't need that. We consider to move it from base FileSystemStorageStatistics to a dedicated
HDFS specific StorageStatistics sub-class.
> # We would have to develop an optimized thread-local mechanism to do this, to avoid causing
a performance regression in HDFS stream performance.
> Optionally, it would be better to simply move this to HDFS's existing per-stream {{ReadStatistics}}
for now. As [HDFS-9579] states, ReadStatistics metrics are only accessible via {{DFSClient}}
or {{DFSInputStream}}. Not something that application framework such as MR and Tez can get
to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message