hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@maprtech.com>
Subject Re: Dedicated disk for operating system
Date Thu, 11 Aug 2011 06:13:27 GMT
Actually I was assuming that the entire cluster participates in the
rebalancing.  Repication is not done disk-wise in hadoop but block-wise.

On Wednesday, August 10, 2011, Rajiv Chittajallu <rajive@yahoo-inc.com>
wrote:
> Ted Dunning wrote on 08/10/11 at 10:40:30 -0700:
>>To be specific, taking a 100 node x 10 disk x 2 TB configuration with
drive
>>MTBF of 1000 days, we should be seeing drive failures on average once per
>>day.  With 1G ethernet and 30MB/s/node dedicated to re-replication, it
will
>>just over 10 minutes to restore replication of a single drive and will
take
>>just over 100 minutes to restore replication of an entire machine.
>
> You are assuming that only one good node is used to restore replication
for
> all the blocks on the failed drive. Which is very unlikely. With
> replication factor of 3, you will have at least 2 nodes to choose from
> in the worst case and much more in a standard cluster.
>
> And when you are having more spindles, 6+, one would probably consider
> using the second GigE port, which is standard on most of the commodity
> gear out there.
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message