hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: How 'commodity' is 'commodity'
Date Tue, 29 Sep 2009 15:15:11 GMT
Edward Capriolo wrote:

> In hadoop terms commodity means "Not super computer". If you look
> around most large deployments have DataNodes with duel quad core
> processors 8+GB ram and numerous disks, that is hardly the PC you find
> under your desk.

I have 4 cores and 6GB RAM, but only one HDD on the desk. That ram gets 
sucked up by the memory hogs: IDE, firefox and VMWare. The fact that 
VMWare uses less RAM to host an entire OS image than firefox shows that 
you can get away with virtualised work, or that firefox is overweight.

18862        20   0 2139m 1.4g  48m S    0 25.0  46:57.81 java 

12822        20   0 1050m 476m  18m S    3  8.1  53:34.39 firefox 

13037        20   0  949m 386m 365m S   11  6.5 206:34.70 vmware-vmx 

20932        20   0  688m 374m  11m S    0  6.4   2:05.18 java

> For example, we had a dev clusters is a very modest setup, 5 HP DL
> 145. Duel Core 4 GB RAM, 2 SATA DISKS.
> 
> I did not do any elaborate testing, but i found that:
> 
> !!!!One!!!! DL180G5 (2x Quad Code) (8GB ram) 8 SATA disk crushed the 5
> node cluster. So it is possible but it might be more administrative
> work then it is worth.

Interesting. Is this CPU or IO intensive work?

Mime
View raw message