Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: error (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CA+ndhHpUuG=t0fOSK_L2QjN-ietEq-gauUb7=S0dz014xK_wzw@mail.gmail.com>
References: 
 <CA+ndhHpUuG=t0fOSK_L2QjN-ietEq-gauUb7=S0dz014xK_wzw@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Fri, 26 Apr 2013 11:00:26 -0700
Message-ID: 
 <CAND0qzsML82zd0Z33+8BbE5ZgHL5TxD7SYXX=Ut6pMJWG=SK8w@mail.gmail.com>
Subject: Re: M/R job optimization
To: "common-user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=089e013d10107f5f4404db4750e7

--089e013d10107f5f4404db4750e7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Have you checked the logs?

Is there a task that is taking a long time?  What is that task doing?

There are two basic possibilities:

a) you have a skewed join like the other Ted mentioned.  In this case, the
straggler will be seen to be working on data.

b) you have a hung process.  This can be more difficult to diagnose, but
indicates that there is a problem with your cluster.


On Fri, Apr 26, 2013 at 2:21 AM, Han JU <ju.han.felix@gmail.com> wrote:

> Hi,
>
> I've implemented an algorithm with Hadoop, it's a series of 4 jobs. My
> questionis that in one of the jobs, map and reduce tasks show 100% finish=
ed
> in about 1m 30s, but I have to wait another 5m for this job to finish.
> This job writes about 720mb compressed data to HDFS with replication
> factor 1, in sequence file format. I've tried copying these data to hdfs,
> it takes only < 20 seconds. What happened during this 5 more minutes?
>
> Any idea on how to optimize this part?
>
> Thanks.
>
> --
> *JU Han*
>
> UTC   -  Universit=E9 de Technologie de Compi=E8gne
> *     **GI06 - Fouille de Donn=E9es et D=E9cisionnel*
>
> +33 0619608888
>

--089e013d10107f5f4404db4750e7
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Have you checked the logs?<div><br></div><div style>Is the=
re a task that is taking a long time? =A0What is that task doing?</div><div=
 style><br></div><div style>There are two basic possibilities:</div><div st=
yle>

<br></div><div style>a) you have a skewed join like the other Ted mentioned=
. =A0In this case, the straggler will be seen to be working on data.</div><=
div style><br></div><div style>b) you have a hung process. =A0This can be m=
ore difficult to diagnose, but indicates that there is a problem with your =
cluster.</div>

<div style><br></div></div><div class=3D"gmail_extra"><br><br><div class=3D=
"gmail_quote">On Fri, Apr 26, 2013 at 2:21 AM, Han JU <span dir=3D"ltr">&lt=
;<a href=3D"mailto:ju.han.felix@gmail.com" target=3D"_blank">ju.han.felix@g=
mail.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi,<div><br></div><div>I=
9;ve implemented an algorithm with Hadoop, it&#39;s a series of 4 jobs. My =
questionis that in one of the jobs, map and reduce tasks show 100% finished=
 in about 1m 30s, but I have to wait another 5m for this job to finish.</di=
v>


<div>This job writes about 720mb compressed data to HDFS with replication f=
actor 1, in sequence file format. I&#39;ve tried copying these data to hdfs=
, it takes only &lt; 20 seconds. What happened during this 5 more minutes?<=
/div>


<div><br></div><div>Any idea on how to optimize this part?=A0<br clear=3D"a=
ll"><div><br></div><div>Thanks.</div><span class=3D"HOEnZb"><font color=3D"=
#888888"><div><br></div>-- <br><div dir=3D"ltr"><font face=3D"verdana, sans=
-serif"><b>JU Han</b></font><div>

<br></div>
<div><div><div><div><div><div><div><span style=3D"font-size:13px">UTC=A0=A0=
 - =A0<font face=3D"verdana, sans-serif">Universit=E9 de Technologie de Com=
pi=E8gne</font></span></div></div></div><div><div><div><div><i>=A0=A0=A0=A0=
 </i><i><i style=3D"font-family:verdana,sans-serif">GI06 - Fouille de Donn=
=E9es et D=E9cisionnel</i></i></div>


</div></div></div></div><div><br></div><div><div><font face=3D"verdana, san=
s-serif"><a href=3D"tel:%2B33%200619608888" value=3D"+33619608888" target=
=3D"_blank">+33 0619608888</a></font></div></div></div></div></div></div>
</font></span></div></div>
</blockquote></div><br></div>

--089e013d10107f5f4404db4750e7--