hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-324) Generate a network infrastructre map
Date Sat, 19 Jul 2014 18:58:40 GMT

     [ https://issues.apache.org/jira/browse/HDFS-324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Allen Wittenauer resolved HDFS-324.

    Resolution: Duplicate

This is part of 289.

> Generate a network infrastructre map
> ------------------------------------
>                 Key: HDFS-324
>                 URL: https://issues.apache.org/jira/browse/HDFS-324
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kai Mosebach
> Assuming an inhomogeneous network it might be sensible to not only collect metrics about
to and from the nodes but also
> - collect latency between nodes
> - network io between nodes
> - daytime depending network io between nodes (i.e. network is much slower in the morning
time and in the early evening time)
> - network failures
> Assuming the we collect aging information over a period of time, it would allow us to
create a "learning cloud" in senses of its infrastructure. From this a network map we can
> - see subclusters of high speed linked nodes*
> - see unreliable connections between nodes
> - see heavily used links over the time
> These information can be refed into the DFS (for data distribution as well as for the
balancer) logic to increase its reliability and its performance a lot.
> *Note : This could already be managed with the rack awareness, but the aging approach
would make this much more fine grained and in an automatic manner.

This message was sent by Atlassian JIRA

View raw message