hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tim robertson <timrobertson...@gmail.com>
Subject Re: Hardware - please sanity check?
Date Thu, 02 Apr 2009 15:33:27 GMT
Thanks Miles,

Thus far most of my work has been on EC2 large instances and *mostly*
my code is not memory intensive (I sometimes do joins against polygons
and hold Geospatial indexes in memory, but am aware of keeping things
within the -Xmx for this).
I am mostly  looking to move routine data processing and
transformation (lots of distinct, count and group by operations) off a
chunky mysql DB (200million rows and growing) which gets all locked

We have gigabit switches.



On Thu, Apr 2, 2009 at 4:15 PM, Miles Osborne <miles@inf.ed.ac.uk> wrote:
> make sure you also have a fast switch, since you will be transmitting
> data across your network and this will come to bite you otherwise
> (roughly, you need one core per hadoop-related job, each mapper, task
> tracker etc;  the per-core memory may be too small if you are doing
> anything memory-intensive.  we have 8-core boxes with 50 -- 33 GB RAM
> and 8 x 1 TB disks on each one;  one box however just has 16 GB of RAM
> and it routinely falls over when we run jobs on it)
> Miles
> 2009/4/2 tim robertson <timrobertson100@gmail.com>:
>> Hi all,
>> I am not a hardware guy but about to set up a 10 node cluster for some
>> processing of (mostly) tab files, generating various indexes and
>> researching HBase, Mahout, pig, hive etc.
>> Could someone please sanity check that these specs look sensible?
>> [I know 4 drives would be better but price is a factor (second hand
>> not an option, hosting is not either as there is very good bandwidth
>> provided)]
>> Something along the lines of:
>> Dell R200 (8GB is max memory)
>> Quad Core Intel® Xeon® X3360, 2.83GHz, 2x6MB Cache, 1333MHz FSB
>> 8GB Memory, DDR2, 800MHz (4x2GB Dual Ranked DIMMs)
>> 2x 500GB 7.200 rpm 3.5-inch SATA Hard Drive
>> Dell R300 (can be expanded to 24GB RAM)
>> Quad Core Intel® Xeon® X3363, 2.83GHz, 2x6M Cache, 1333MHz FS
>> 8GB Memory, DDR2, 667MHz (2x4GB Dual Ranked DIMMs)
>> 2x 500GB 7.200 rpm 3.5-inch SATA Hard Drive
>> If there is a major flaw please can you let me know.
>> Thanks,
>> Tim
>> (not a hardware guy ;o)
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.

View raw message