Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type;
  b=SJlJmQfb+U9XP3kJi3f4vUN5JnMYdB9URaCvfX2ZcmarqsGhPbs2I3T/98uaRfXs67NRNdh1NdzxSrFn9XfsKKtp29805vPPbQUd+c+TdnKDo5oDA7fuBRo+uV002PjwS51+ZqBfl6z2GiYMUJ7yV15Ah3JgU9Vgh6A/Z9e1K30=;
Message-ID: <881015.94450.qm@web33508.mail.mud.yahoo.com>
Date: Thu, 13 Jan 2011 08:51:22 -0800 (PST)
From: Raj V <rajvish@yahoo.com>
Reply-To: Raj V <rajvish@yahoo.com>
Subject: Re: TeraSort question.
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-756794436-1294937482=:94450"

--0-756794436-1294937482=:94450
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Steve=0A=0ALet me plot the graphs for all the nodes. I picked up 6 random n=
odes out oif 480 and 2 of these were really busy and the otehr 4 were idle.=
 Either that makes me very lucky or the cluster was underutilized.=0A=0AI w=
ould have found it acceptable if different nodes were utilized in different=
 ways, but in my case , 2 nodes had serious CPU , Network and Disk activity=
 and others=A0 were completely idle.=0A=0A=0A=0A=0A=0A=0A=0A=0AFrom: Steve =
Loughran <stevel@apache.org>=0ATo: common-user@hadoop.apache.org=0ACc: =0AS=
ent: Thursday, January 13, 2011 3:05 AM=0ASubject: Re: TeraSort question.=
=0A=0AOn 11/01/11 16:40, Raj V wrote:=0A> Ted=0A> =0A> =0A> Thanks. I have =
all the graphs I need that include, map reduce timeline, system activity fo=
r all the nodes when the sort was running. I will publish them once I have =
them in some presentable format.,=0A> =0A> For legal reasons, I really don'=
t want to send the complete job histiory files.=0A> =0A> My question is sti=
ll this. When running terasort, would the CPU, disk and network utilization=
 of all the nodes be more or less similar or completely different.=0A=0AThe=
y can be different. The JT pushes out work to machines when they report in,=
 some may get more work than others, so generate more local data. This will=
 have follow-on consequences. In a live system things are different as the =
work tends to follow the data, so machines with (or near) the data you need=
 get the work.=0A=0AIt's a really hard thing to say "is the cluster working=
 right", when bringing it up, everyone is really guessing about expected pe=
rformance.=0A=0A-Steve
--0-756794436-1294937482=:94450--