Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <16711AE3-EFCE-4D46-8403-14296245B192@connexity.com>
References: <16711AE3-EFCE-4D46-8403-14296245B192@connexity.com>
Date: Sat, 23 Jan 2016 14:00:50 +0200
Message-ID: 
 <CAHs8FG23PHDb=YOf6MgXbmG1qK9DeHrhMeeUTnkh3Mj+dZvrBg@mail.gmail.com>
Subject: Re: NodeManager High CPU due to high GC
From: Daniel Haviv <danielrulez@gmail.com>
To: Randy Fox <rfox@connexity.com>
Cc: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a11c33f8042e6170529ff16b4

--001a11c33f8042e6170529ff16b4
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Randy,
How much cores do you have on your machines and how much did you allocate
for Yarn?

Daniel

On Saturday, 23 January 2016, Randy Fox <rfox@connexity.com> wrote:

> Hi,
>
> We just upgraded to using Yarn on Hadoop 2.6.0 =E2=80=93 CDH5.4.5
> We are running a large job =E2=80=93 200K mappers, 100K reducers and we c=
an=E2=80=99t get
> through the shuffle phase.  The node managers are 800% cpu and high GC.
> The reducers get socket timouts after 1.5 hours of running and only getti=
ng
> a few percent of the data from the mappers.  This job took about 30 hours
> total 12 in mappers on MRv1 with no issues.
>
> I have looked for configs that might help or issues filed and anyone that
> has seen this and I have come up with nothing.
> Anyone have ideas on things to try or explain why the node managers are i=
n
> GC hell and why the data is just not flowing from mappers to reducers?
>
> Thanks in advanced,
>
> Randy
>

--001a11c33f8042e6170529ff16b4
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Randy,<div>How much cores do you have on your machines and how much did =
you allocate for Yarn?</div><div><br></div><div>Daniel<span></span><br><br>=
On Saturday, 23 January 2016, Randy Fox &lt;<a href=3D"mailto:rfox@connexit=
y.com">rfox@connexity.com</a>&gt; wrote:<br><blockquote class=3D"gmail_quot=
e" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


<div style=3D"word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-fam=
ily:&#39;Times New Roman&#39;,sans-serif">
<div>
<div>
<div>Hi,</div>
<div><br>
</div>
<div>We just upgraded to using Yarn on Hadoop 2.6.0 =E2=80=93 CDH5.4.5</div=
>
<div>We are running a large job =E2=80=93 200K mappers, 100K reducers and w=
e can=E2=80=99t get through the shuffle phase.=C2=A0 The node managers are =
800% cpu and high GC.=C2=A0 The reducers get socket timouts after 1.5 hours=
 of running and only getting a few percent of the data from
 the mappers.=C2=A0 This job took about 30 hours total 12 in mappers on MRv=
1 with no issues.</div>
<div><br>
</div>
<div>I have looked for configs that might help or issues filed and anyone t=
hat has seen this and I have come up with nothing.</div>
<div>Anyone have ideas on things to try or explain why the node managers ar=
e in GC hell and why the data is just not flowing from mappers to reducers?=
</div>
<div><br>
</div>
<div>Thanks in advanced,</div>
<div><br>
</div>
<div>Randy</div>
<div>
<div></div>
</div>
</div>
</div>
</div>

</blockquote></div>

--001a11c33f8042e6170529ff16b4--