hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Nelson <had...@enigmasupercomputing.com>
Subject Hard drives dropping like flies
Date Tue, 21 Jul 2009 05:44:22 GMT
I have a question that I think I already know the answer to but I would 
like verification.  I have a demo cluster comprised of two master nodes 
and eight slaves (all 1x1.2 Ghz cpu / 1 Gig Ram / 1x250 Gig Sata 7200 
rpm hard drives).  I'm running small MR processes, about 100-200Gigs of 
total data that take about 1-2 hours to process. These small processes 
seem to work fine. However, I'm starting to run larger processes on them 
(5-8 hour processes with 200-300Gigs of data) and the hard drives keep 
dieing. I know I'm not running out of space, the hard drives really are 
crashing under the load.  I don't think it's over heating because the 
server room temp is a constant 68-72 degrees.  I'm running under the 
default configuration, 2 maps + 2 reduces per node.  I suspect that 
since I only have one hard drive per node it is almost continuously 
writing to at least four different files (on the same HD) and its just 
thrashing the read/write head and the motor. I assumed my first batch of 
HD's were just bad but I've just had 4 more brand new drives fail within 
a week so I think I'm pushing it too hard.

If this is the problem, do I need to be sure and place my dfs data and 
mapred data directories onto separate physical drives?  If I upgrade to 
some dell servers with dual core and 4-5HD's per node will hadoop take 
care of balancing the load between hard drives (dfs/mapred/scratch storage)?

Tim Nelson

View raw message