hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-324) Generate a network infrastructre map
Date Sat, 19 Jul 2014 18:58:40 GMT

     [ https://issues.apache.org/jira/browse/HDFS-324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Allen Wittenauer resolved HDFS-324.
-----------------------------------

    Resolution: Duplicate

This is part of 289.

> Generate a network infrastructre map
> ------------------------------------
>
>                 Key: HDFS-324
>                 URL: https://issues.apache.org/jira/browse/HDFS-324
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kai Mosebach
>
> Assuming an inhomogeneous network it might be sensible to not only collect metrics about
to and from the nodes but also
> - collect latency between nodes
> - network io between nodes
> - daytime depending network io between nodes (i.e. network is much slower in the morning
time and in the early evening time)
> - network failures
> Assuming the we collect aging information over a period of time, it would allow us to
create a "learning cloud" in senses of its infrastructure. From this a network map we can
> - see subclusters of high speed linked nodes*
> - see unreliable connections between nodes
> - see heavily used links over the time
> These information can be refed into the DFS (for data distribution as well as for the
balancer) logic to increase its reliability and its performance a lot.
> *Note : This could already be managed with the rack awareness, but the aging approach
would make this much more fine grained and in an automatic manner.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message