Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
From: Nicolae Marasoiu <nicolae.marasoiu@adswizz.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: yarn does not allocate enough tasks/containers to my available node
Thread-Topic: yarn does not allocate enough tasks/containers to my available
 node
Thread-Index: AQHRJgcDECGGQhiNyk+WTOHuy+Xapw==
Date: Mon, 23 Nov 2015 15:58:43 +0000
Message-ID: 
 <BY2PR11MB08243D197441D36254E3627F8F070@BY2PR11MB0824.namprd11.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: adswizz.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_BY2PR11MB08243D197441D36254E3627F8F070BY2PR11MB0824namp_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Nov 2015 15:58:43.0106
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: d1ae4f29-fb22-47cb-b597-403b799a1628
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR11MB0823

--_000_BY2PR11MB08243D197441D36254E3627F8F070BY2PR11MB0824namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi,


Tasks are allocated to my nodes by memory.

Initially they are allocated ok across the cluster.

After a while, one of the nodes does not receive new tasks fast enough: it =
gets to 0 tasks and from time to time I see it having 1 task which it finis=
hed in seconds.


It is true that I currently have a problem of many small input files.

And probably the fact that the nodes are oversubscribed in cpu by a factor =
of 2-3 (according to load average) is not helping.


But 1. why does yarn not able to bulk allocate some 4 tasks on the idle nod=
e at once (not one by one), and 2. why yarn is slow in allocating tasks? (I=
 understand that allocating a new task/container in a few seconds may/may n=
ot be considered slow).


Pls advise,

Nicu

--_000_BY2PR11MB08243D197441D36254E3627F8F070BY2PR11MB0824namp_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
</head>
<body dir=3D"ltr">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;back=
ground-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi,</p>
<p><br>
</p>
<p>Tasks are allocated to my nodes by memory.</p>
<p>Initially they are allocated ok across the cluster.</p>
<p>After a while, one of the nodes does not receive new tasks fast enough: =
it gets to 0 tasks and from time to time I see it having 1 task which it fi=
nished in seconds.</p>
<p><br>
</p>
<p>It is true that I currently have a problem of many small input files.&nb=
sp;</p>
<p>And probably the fact that the nodes are oversubscribed in cpu by a fact=
or of 2-3 (according to load average) is not helping.</p>
<p><br>
</p>
<p>But 1.&nbsp;why does yarn not able to bulk allocate some 4 tasks on the =
idle node at once (not one by one), and 2. why yarn is slow in allocating t=
asks? (I understand that allocating a new task/container in a few seconds m=
ay/may not be considered slow).</p>
<p><br>
</p>
<p>Pls advise,</p>
<p>Nicu</p>
</div>
</body>
</html>

--_000_BY2PR11MB08243D197441D36254E3627F8F070BY2PR11MB0824namp_--