hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: Question on DFS block placement and 'what is a rack' wrt DFS block placement
Date Tue, 12 Feb 2008 23:31:32 GMT

It isn't popular much anymore, but once upon a time, network topology for
clustering was a big topic.  Since then, switches have gotten pretty fast
and worrying about these things has gone out of fashion a bit other than
something on the level of the current rack-aware locality in Hadoop.

With 4 NIC's, you could replay some history, however, by building what
amounts to a 4-dimensional hyper-torus.  I *think* you could pretty well by
having four parallel two-level switch networks and assign boxes to second
level switches according to a systematic pattern so that you would have
local access to much of the cluster.  As a simple example, suppose that you
have 16 machines M1 through M16, each with two interfaces.  You would have
two top level switches T1 and T2 and each of those would have four second
level switches S1.1 ... S1.4 connected to T1 and S2.1 ... S2.4 connected to
T2.  The machine connectivity on each interface would look like this:

Machine        eth0               eth1
 1                   S1.1               S2.1
 2                   S1.1               S2.2
 3                   S1.1               S2.3
 4                   S1.1               S2.4
 5                   S1.2               S2.1
 6                   S1.2               S2.2
 7                   S1.2               S2.3
 8                   S1.2               S2.4
 9                   S1.3               S2.1
10                  S1.3                S2.2
11                  S1.3                S2.3
12                  S1.3                S2.4
13                  S1.4                S2.1
14                  S1.4                S2.2
15                  S1.4                S2.3
16                  S1.4                S2.4

With this arrangement machine M1 is on a local switch with M2, M3, M4, M5,
M9, and M13 which is twice as many machines as would be local if only one
interface were used.  With four interfaces and four machines on a local
switch, the entire cluster is local and T1 and T2 should see no traffic.  In
a larger cluster, you get the same 16x benefit in locality.

The cost is that your ops guys will kill you if you suggest something this
elaborate.  The wiring between racks alone will make this a nightmare.

On 2/12/08 3:01 PM, "Jason Venner" <jason@attributor.com> wrote:

> In terms of more exotic situations we were discussing having 4 NIC's 1
> for the local subset, 1 for a pair of local subsets, 1 for another pair
> of local subsets, 1 for the backbone.

View raw message