hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Mosebach (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-4082) Generate a network infrastructre map
Date Fri, 05 Sep 2008 15:18:46 GMT
Generate a network infrastructre map

                 Key: HADOOP-4082
                 URL: https://issues.apache.org/jira/browse/HADOOP-4082
             Project: Hadoop Core
          Issue Type: Improvement
          Components: dfs, metrics
            Reporter: Kai Mosebach

Assuming an inhomogeneous network it might be sensible to not only collect metrics about to
and from the nodes but also

- collect latency between nodes
- network io between nodes
- daytime depending network io between nodes (i.e. network is much slower in the morning time
and in the early evening time)
- network failures

Assuming the we collect aging information over a period of time, it would allow us to create
a "learning cloud" in senses of its infrastructure. From this a network map we can

- see subclusters of high speed linked nodes*
- see unreliable connections between nodes
- see heavily used links over the time

These information can be refed into the DFS (for data distribution as well as for the balancer)
logic to increase its reliability and its performance a lot.

*Note : This could already be managed with the rack awareness, but the aging approach would
make this much more fine grained and in an automatic manner.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message