Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
From: gmail <juer.kaiser@gmail.com>
Content-Type: multipart/signed;
 boundary="Apple-Mail=_78374A78-EE7D-40D8-9D51-18024D66CDBB";
 protocol="application/pgp-signature"; micalg=pgp-sha512
Subject: Yarn doesn't start mappers fast enough
Date: Tue, 6 Oct 2015 12:13:39 +0200
Message-Id: <179AFCE3-428F-4311-A461-3259EDF44C8C@gmail.com>
To: user@hadoop.apache.org
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\))


--Apple-Mail=_78374A78-EE7D-40D8-9D51-18024D66CDBB
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_26047C2B-0974-4646-9781-31F921A924FD"


--Apple-Mail=_26047C2B-0974-4646-9781-31F921A924FD
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hallo everyone,

I have a problem with my yarn setup and hope you can help me. I already =
searched for this issue but didn't find anything.

My problem is that yarn doesn=E2=80=99t start new mappers fast enough. =
This results in a poor cluster utilization.

Setup:
 - 8 nodes @64cores+128GB
 - Hadoop version: Hadoop 2.6.0,
 - Standard Terasort of 100GB, input data generated by teragen with two =
mappers

What I see: At most ~40 mappers run at the same time. It looks like the =
rate of starting new mappers and the finishing rate is about the same at =
that point. The avg. processing time of each mapper is about 34-40s. If =
I start a second Terasort at the same time, it also  only runs up to ~40 =
mappers. It seems that 1) yarn correctly detects that it can run more =
but 2) doesn't start new mappers fast enough (1 at a time?).
What I expect: better utilization of all nodes since there are 300+ map =
jobs.

Are there parameters to change this behavior? How can I tell yarn to =
start more instances at the same time?

for completeness:
    - the behavior doesn't change if I use more mappers during teragen.
    - the bahavior doesn't change if I modify the number of nodes.
    - I recompiled Hadoop for 64bit according to =
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/N=
ativeLibraries.html =
<https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/=
NativeLibraries.html>
    - I use an GPFS as backend with the IBM gpfs-connector.

Thanks in advance,
J=C3=BCrgen

--Apple-Mail=_26047C2B-0974-4646-9781-31F921A924FD
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div class=3D"">Hallo everyone,</div><div class=3D""><br =
class=3D""></div><div class=3D"">I have a problem with my yarn setup and =
hope you can help me. I already searched for this issue but didn't find =
anything.</div><div class=3D""><br class=3D""></div><div class=3D"">My =
problem is that yarn doesn=E2=80=99t start new mappers fast enough. This =
results in a poor cluster utilization.</div><div class=3D""><br =
class=3D""></div><div class=3D"">Setup:&nbsp;</div><div class=3D"">&nbsp;-=
 8 nodes @64cores+128GB</div><div class=3D"">&nbsp;- Hadoop version: =
Hadoop 2.6.0,</div><div class=3D"">&nbsp;- Standard Terasort of 100GB, =
input data generated by teragen with two mappers</div><div class=3D""><br =
class=3D""></div><div class=3D"">What I see: At most ~40 mappers run at =
the same time. It looks like the rate of starting new mappers and the =
finishing rate is about the same at that point. The avg. processing time =
of each mapper is about 34-40s. If I start a second Terasort at the same =
time, it also &nbsp;only runs up to ~40 mappers. It seems that 1) yarn =
correctly detects that it can run more but 2) doesn't start new mappers =
fast enough (1 at a time?).</div><div class=3D"">What I expect: better =
utilization of all nodes since there are 300+ map jobs.</div><div =
class=3D""><br class=3D""></div><div class=3D"">Are there parameters to =
change this behavior? How can I tell yarn to start more instances at the =
same time?</div><div class=3D""><br class=3D""></div><div class=3D"">for =
completeness:</div><div class=3D"">&nbsp; &nbsp; - the behavior doesn't =
change if I use more mappers during teragen.</div><div class=3D"">&nbsp; =
&nbsp; - the bahavior doesn't change if I modify the number of =
nodes.</div><div class=3D"">&nbsp; &nbsp; - I recompiled Hadoop for =
64bit according to&nbsp;<a =
href=3D"https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-=
common/NativeLibraries.html" =
class=3D"">https://hadoop.apache.org/docs/current/hadoop-project-dist/hado=
op-common/NativeLibraries.html</a></div><div class=3D"">&nbsp; &nbsp; - =
I use an GPFS as backend with the IBM gpfs-connector.</div><div =
class=3D""><br class=3D""></div><div class=3D"">Thanks in =
advance,</div><div class=3D"">J=C3=BCrgen</div></body></html>=

--Apple-Mail=_26047C2B-0974-4646-9781-31F921A924FD--

--Apple-Mail=_78374A78-EE7D-40D8-9D51-18024D66CDBB
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQQcBAEBCgAGBQJWE57ZAAoJEJU3ZLZYFRWm4W4gAJlmad48DfUw6teAA0dIx1ns
rX8ucV9/lu8hqegNr3wg736hZbqL73Vi/bcQ/C253HGuiEIMhrmTspoWcQYKvI3d
qxoy9r4/UMqxfoJd4dJaAfI9hZ5P3DAmND9looUBsnYSnh+X/U0AZJ0vPnb2vr0L
sN6MXlohH5p0PqRo88VNQw5HKj/NevhyyxlVYKRmV36KmMrWwTFywwqTLT2MVEsJ
cd/EkHWGe0UoTn1uwjycNAA+Dx21NHz35ApbJxRZEWWx3BfBLpafIz3M5hBSrBVq
5H1HyNJBOSGqnHPndaf0twDiFOZCSAFJPoO8YYzt5TWJ9BI//cXQLfpxdyhjwBig
fVMGgXsBMzpvwu2xcyiGRmqk3dm/V6Y8TN4TKhDgL6n4z8hQbjuLcwzpwquWRlMf
eWxGbG9t/UV5aSWySJpBW+SWn48MD1LT350+EGX+waJqZ3vcXnCMp/FIlAdmgYkk
AYUk3iuHVZFW7MjUD6Wa/+M7tLH7JeW9KJ6OnQuNmfkzwe7UqG8FrQXmCM81Ngt1
zfHBngsWkJpIqgTAV6rhRlTS/USVVkQPzsz7rU/ei9DI9UYYeBmHb0V2F3hQWrH8
s6thA+o+kqpQ8XFB5FkrJ2arkKX3m41abnz81OC8xJ5OPixUvP74SZ8MQ2CsnHw7
CR3ih6hYVndyg12DsiR96RzNhhceY69fuxqKhXFUXym2LGZ+1ntyseQhqnGoWUMs
zbi2WLzNotKO0BaqL46oGsnJGqAPu7SlNpO/XcyuFdxYdxpuXiRRXURUyYBg3wvb
pnP/8XOGLxMcpFHXgtCgaptqIhRAff/cFOGD2HjRLLpYNnZtfxKTU2oiZ8/yci1J
Y5SBu7ApvxK2WCbRqF2VCcERf8PCwE2HSGJGVK4mflwsztMiIaCtep9dp9EdUENc
medxM8ceaut5BQKNFycbof85gdsp2VcYQwtDf2Sw34s2BuXvIEMtMeqfhG8R/Ym+
hoiwvKeH7iufWKP/wtcUpdSH7lPT2q7VGsUaPOEqSH3qP38CC8UIytZ0GzVUeNA1
UoOXd/I90iQTkuoSONbmMfm+kBln1brCeNoQixp9uM2BZtHwPv0U7alqglBLAUI6
ppKkBRbjkyMF6+ozCI02BbUpZK/x8SyJVz6dQZr3f+EZeh1oJtB+dqqH2U24okzc
OT5+foOOAhW3TfU4mLl3B3czdabQSy7TfiujzswWUXWavfLl8QiYA3wgmZdLhXUj
2Sy4szeXsb7aH8dTVC/6TZbyjtv3x/4mmHaOj1/mUz8UrU8vvGNSOFxDSCFiBJMK
nE9pBuEniB5TR2ZuvDX2oKepV02EXERRgrOu6nsY1nGExCr9vFcMLKlUT4/+y5A=
=tHfY
-----END PGP SIGNATURE-----

--Apple-Mail=_78374A78-EE7D-40D8-9D51-18024D66CDBB--