Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of patrickangeles@gmail.com
 designates 209.85.210.194 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=gfqq9KuJBWeAb8PrzCq6TUNPFOJ+zbIxFM56eHNT1ibRYUahu6wROoN36tOwcNdcjl
         TzOCm39pQbX2A4SIhwD47PbpVSN1xDFUjgFCporFZCVqoeqtrucWEPCq2nCJ9M4hheN8
         /RD8xjxgvxa3kiTAxMI+ER0YoqiG4vtFVkKDU=
MIME-Version: 1.0
In-Reply-To: <32120a6a0910150834x2016eeb1k10b9c4e0c8794248@mail.gmail.com>
References: <4AD59040.8030008@opera.com>
	 <32120a6a0910140202x5cd65b47vbc00e1a92d578bc7@mail.gmail.com>
	 <4AD59B93.3050905@opera.com>
	 <32120a6a0910140246x1b674fffj7eb1917ec2bf29e4@mail.gmail.com>
	 <4AD5B04A.4030700@opera.com>
	 <45f85f70910141017q716d322vceac08dbd6e3f880@mail.gmail.com>
	 <4AD6EC33.1080308@opera.com>
	 <32120a6a0910150834x2016eeb1k10b9c4e0c8794248@mail.gmail.com>
Date: Thu, 15 Oct 2009 11:52:05 -0400
Message-ID: <a6906bde0910150852k217eef61re1f02faee0b5c33@mail.gmail.com>
Subject: Re: Hardware performance from HADOOP cluster
From: Patrick Angeles <patrickangeles@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e640d3fed677b70475fb42b4

--0016e640d3fed677b70475fb42b4
Content-Type: text/plain; charset=ISO-8859-1

Hi Tim,
I assume those are single proc machines?

I got 649 secs on 70GB of data for our 7-node cluster (~11 mins), but we
have dual quad Nehalems (2.26Ghz).

On Thu, Oct 15, 2009 at 11:34 AM, tim robertson
<timrobertson100@gmail.com>wrote:

> Hi Usmam,
>
> So on my 10 node cluster (9 DN) with 4 maps and 4 reduces (I plan on
> high memory jobs so picked 4 only)
> [9 DN of Dell R300: 2.83G Quadcore (2x6MB cache), 8G RAM and 2x500G SATA
> drives]
>
> Using your template for stats, I get the following with no tuning:
>
> GENERATE RANDOM DATA
> Wrote out 90GB of random binary data:
> Map output records=9198009
> The job took 350 seconds. (approximately: 6 minutes)
>
> SORT RANDOM GENERATED DATA
> Map output records= 9197821
> Reduce input records=9197821
> The job took 2176 seconds. (approximately: 36mins).
>
> So pretty similar to your initial benchmark.  I will tune it a bit
> tomorrow and rerun.
>
> If you spent time tuning your cluster and it was successful, please
> can you share your config?
>
> Cheers,
> Tim
>
>
>
>
>
> On Thu, Oct 15, 2009 at 11:32 AM, Usman Waheed <usmanw@opera.com> wrote:
> > Hi Todd,
> >
> > Some changes have been applied to the cluster based on the documentation
> > (URL) you noted below,
> > like file descriptor settings and io.file.buffer.size. I will check out
> the
> > other settings which I haven't applied yet.
> >
> > My map/reduce slot settings from my hadoop-site.xml and
> hadoop-default.xml
> > on all nodes in the cluster.
> >
> > _*hadoop-site.xml
> > *_mapred.tasktracker.task.maximum = 2
> > mapred.tasktracker.map.tasks.maximum = 8
> > mapred.tasktracker.reduce.tasks.maximum = 8
> > _*
> > hadoop-default.xml
> > *_mapred.map.tasks = 2
> > mapred.reduce.tasks = 1
> >
> > Thanks,
> > Usman
> >
> >
> >> This seems a bit slow for that setup (4-5 MB/sec/node sorting). Have
> >> you changed the configurations at all? There are some notes on this
> >> blog post that might help your performance a bit:
> >>
> >>
> >>
> http://www.cloudera.com/blog/2009/03/30/configuration-parameters-what-can-you-just-ignore/
> >>
> >> How many map and reduce slots did you configure for the daemons? If
> >> you have Ganglia installed you can usually get a good idea of whether
> >> you're using your resources well by looking at the graphs while
> >> running a job like this sort.
> >>
> >> -Todd
> >>
> >> On Wed, Oct 14, 2009 at 4:04 AM, Usman Waheed <usmanw@opera.com> wrote:
> >>
> >>>
> >>> Here are the results i got from my 4 node cluster (correction i noted 5
> >>> earlier). One of my nodes out of the 4 is a namenode+datanode both.
> >>>
> >>> GENERATE RANDOM DATA
> >>> Wrote out 40GB of random binary data:
> >>> Map output records=4088301
> >>> The job took 358 seconds. (approximately: 6 minutes).
> >>>
> >>> SORT RANDOM GENERATED DATA
> >>> Map output records=4088301
> >>> Reduce input records=4088301
> >>> The job took 2136 seconds. (approximately: 35 minutes).
> >>>
> >>> VALIDATION OF SORTED DATA
> >>> The job took 183 seconds.
> >>> SUCCESS! Validated the MapReduce framework's 'sort' successfully.
> >>>
> >>> It would be interesting to see what performance numbers others with a
> >>> similar setup have obtained.
> >>>
> >>> Thanks,
> >>> Usman
> >>>
> >>>
> >>>>
> >>>> I am setting up a new cluster of 10 nodes of 2.83G Quadcore (2x6MB
> >>>> cache), 8G RAM and 2x500G drives, and will do the same soon.  Got some
> >>>> issues though so it won't start up...
> >>>>
> >>>> Tim
> >>>>
> >>>>
> >>>> On Wed, Oct 14, 2009 at 11:36 AM, Usman Waheed <usmanw@opera.com>
> wrote:
> >>>>
> >>>>
> >>>>>
> >>>>> Thanks Tim, i will check it out and post my results for comments.
> >>>>> -Usman
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> Might it be worth running the http://wiki.apache.org/hadoop/Sortand
> >>>>>> posting your results for comment?
> >>>>>>
> >>>>>> Tim
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Oct 14, 2009 at 10:48 AM, Usman Waheed <usmanw@opera.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> Is there a way to tell what kind of performance numbers one can
> >>>>>>> expect
> >>>>>>> out
> >>>>>>> of their cluster given a certain set of specs.
> >>>>>>>
> >>>>>>> For example i have 5 nodes in my cluster that all have the
> following
> >>>>>>> hardware configuration(s):
> >>>>>>> Quad Core 2.0GHz, 8GB RAM, 4x1TB disks and are all on the same
> rack.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Usman
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
> >
>

--0016e640d3fed677b70475fb42b4--