hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dejan Diklic <de...@attributor.com>
Subject Re: Hard drives dropping like flies
Date Tue, 21 Jul 2009 15:02:44 GMT
More drives will certainly help in lots of ways but no new drive should fail
within a week. I'd assume you have either power or heating issues.

Dejan


On Mon, Jul 20, 2009 at 10:44 PM, Tim Nelson <
hadoop@enigmasupercomputing.com> wrote:

> I have a question that I think I already know the answer to but I would
> like verification.  I have a demo cluster comprised of two master nodes and
> eight slaves (all 1x1.2 Ghz cpu / 1 Gig Ram / 1x250 Gig Sata 7200 rpm hard
> drives).  I'm running small MR processes, about 100-200Gigs of total data
> that take about 1-2 hours to process. These small processes seem to work
> fine. However, I'm starting to run larger processes on them (5-8 hour
> processes with 200-300Gigs of data) and the hard drives keep dieing. I know
> I'm not running out of space, the hard drives really are crashing under the
> load.  I don't think it's over heating because the server room temp is a
> constant 68-72 degrees.  I'm running under the default configuration, 2 maps
> + 2 reduces per node.  I suspect that since I only have one hard drive per
> node it is almost continuously writing to at least four different files (on
> the same HD) and its just thrashing the read/write head and the motor. I
> assumed my first batch of HD's were just bad but I've just had 4 more brand
> new drives fail within a week so I think I'm pushing it too hard.
>
> If this is the problem, do I need to be sure and place my dfs data and
> mapred data directories onto separate physical drives?  If I upgrade to some
> dell servers with dual core and 4-5HD's per node will hadoop take care of
> balancing the load between hard drives (dfs/mapred/scratch storage)?
>
> Regards,
> Tim Nelson
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message