Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID;
  b=VD8Ij180nPDwK6mMFusn3j7BYAQxl2P1kZ0xEEnF4FV9JMGDwFvXGXXJzNi6DTpubBa2TT8nZqjeVs+Fi7z8ozH8C3vrR0S0WYnMahc9YQbqK9/8YDXvA/kHbwkKJzNkMhBZtj+kyObLuTBQJE79fr1kgffEHyW/OsX2D/Rj7Sw=;
Date: Tue, 11 Dec 2007 12:43:20 -0800 (PST)
From: Chris Fellows <chrisc_fellows@yahoo.com>
Subject: Re: commodity vs. high perf machines: which would you rather
To: hadoop-user@lucene.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-169910363-1197405800=:61759"
Message-ID: <647793.61759.qm@web81815.mail.mud.yahoo.com>

--0-169910363-1197405800=:61759
Content-Type: text/plain; charset=us-ascii

All of the answers to this thread were critically helpful for management and those trying to understand hadoop and the opportunities. And what kind of hardware we should be looking at.

Does this belong in the FAQ?

Thanks

----- Original Message ----
From: Ted Dunning <tdunning@veoh.com>
To: hadoop-user@lucene.apache.org
Sent: Wednesday, November 7, 2007 4:35:58 PM
Subject: Re: commodity vs. high perf machines: which would you rather


For me, I have three configurations available.

A) database class machine with many (>10) fast SAS drives and >10GB
 memory,
dual or quad x quad core cpu's.  Let's say that this costs about 20K$.

B) generic productiion machine with 2 x 250GB SATA drives, 4-12GB RAM,
 dual
x dual core CPU's (=Dell 1950).  Cost is about 2K$.

C) POS beige box machine with 2 x SATA drives of variable size, 4 GB
 RAM,
single dual core CPU.  Cost is about 1K$.

For a $50K budget, I would take 25x(b) over 50x(c) due to simpler and
smaller admin issues even though cost/performance would be nominally
 about
the same.  I would avoid 2x(a) like the plague.


On 11/7/07 11:56 AM, "Chris Fellows" <chrisc_fellows@yahoo.com> wrote:

> Hello,
> 
> Much of the hadoop documentation speaks to large clusters of
 commodity
> machines. There is a debate on our end about which would be better: a
 small
> number of high performance machines (2 boxes with 4 quad core
 processors) or X
> number of commodity machines. I feel that disk I/O might be the
 bottle neck
> with the 2 high perf machines (though I did just read in the FAQ
 about being
> able to split the dfs-data across multiple drives).
> 
> So this is a "which would rather" question. If you were setting up a
 cluster
> of machines to perform data rollups/aggregation (and other mapred
 tasks) on
> files in the .25-1TB size, which would rather have:
> 
> 1. 2 4 quad core machines with your choice on RAM and number of
 drives
> 2. 10 (or more) commodity machines (as defined on the hadoop wiki)
> 
> And of course a "why?" would be very helpful.
> 
> Thanks!
> 


--0-169910363-1197405800=:61759--