Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of sandy.ryza@cloudera.com
 designates 209.85.220.54 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOTkr-c74LQ4s-YWAdDUUWqyCLgoFq3MVnkGGg964jzPpriTKg@mail.gmail.com>
References: 
 <CAOTkr-c74LQ4s-YWAdDUUWqyCLgoFq3MVnkGGg964jzPpriTKg@mail.gmail.com>
Date: Thu, 3 Oct 2013 10:03:55 -0700
Message-ID: 
 <CACBYxKJJjoy51gG8PZQs0UdO1o6Ug-bR1_OvuyLJWo-=afPB8A@mail.gmail.com>
Subject: Re: Non data-local scheduling
From: Sandy Ryza <sandy.ryza@cloudera.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b16013dc5650504e7d92beb

--047d7b16013dc5650504e7d92beb
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Andre,

Try setting yarn.scheduler.capacity.node-locality-delay to a number between
0 and 1.  This will turn on delay scheduling - here's the doc on how this
works:

For applications that request containers on particular nodes, the number of
scheduling opportunities since the last container assignment to wait before
accepting a placement on another node. Expressed as a float between 0 and
1, which, as a fraction of the cluster size, is the number of scheduling
opportunities to pass up. The default value of -1.0 means don't pass up any
scheduling opportunities.

-Sandy


On Thu, Oct 3, 2013 at 9:57 AM, Andr=E9 Hacker <andrephacker@gmail.com> wro=
te:

> Hi,
>
> I have a 25 node cluster, running hadoop 2.1.0-beta, with capacity
> scheduler (default settings for scheduler) and replication factor 3.
>
> I have exclusive access to the cluster to run a benchmark job and I wonde=
r
> why there are so few data-local and so many rack-local maps.
>
> The input format calculates 44 input splits and 44 map tasks, however, it
> seems to be random how many of them are processed data locally. Here the
> counters of my last tries:
>
> data-local / rack-local:
> Test 1: data-local:15 rack-local: 29
> Test 2: data-local:18 rack-local: 26
>
> I don't understand why there is not always 100% data local. This should
> not be a problem since the blocks of my input file are distributed over a=
ll
> nodes.
>
> Maybe someone can give me a hint.
>
> Thanks,
> Andr=E9 Hacker, TU Berlin
>

--047d7b16013dc5650504e7d92beb
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Andre,<div><br></div><div style>Try setting yarn.schedu=
ler.capacity.node-locality-delay to a number between 0 and 1. =A0This will =
turn on delay scheduling - here&#39;s the doc on how this works:</div><div =
style>
<br></div><div style><span style=3D"color:rgb(51,51,51);font-family:Verdana=
,Helvetica,Arial,sans-serif;font-size:12px">For applications that request c=
ontainers on particular nodes, the number of scheduling opportunities since=
 the last container assignment to wait before accepting a placement on anot=
her node. Expressed as a float between 0 and 1, which, as a fraction of the=
 cluster size, is the number of scheduling opportunities to pass up. The de=
fault value of -1.0 means don&#39;t pass up any scheduling opportunities.</=
span>=A0</div>
<div style><br></div><div style>-Sandy</div></div><div class=3D"gmail_extra=
"><br><br><div class=3D"gmail_quote">On Thu, Oct 3, 2013 at 9:57 AM, Andr=
=E9 Hacker <span dir=3D"ltr">&lt;<a href=3D"mailto:andrephacker@gmail.com" =
target=3D"_blank">andrephacker@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>Hi,<br><br>I have a 25=
 node cluster, running hadoop 2.1.0-beta, with capacity scheduler (default =
settings for scheduler) and replication factor 3.<br>
<br></div><div>I have exclusive access to the cluster to run a benchmark jo=
b and I wonder why there are so few data-local and so many rack-local maps.=
<br>
<br></div><div>The input format calculates 44 input splits and 44 map tasks=
, however, it seems to be random how many of them are processed data locall=
y. Here the counters of my last tries:<br><br></div><div>data-local / rack-=
local:<br>

</div><div>Test 1: data-local:15 rack-local: 29<br></div><div>Test 2: data-=
local:18 rack-local: 26<br></div><div><br></div><div>I don&#39;t understand=
 why there is not always 100% data local. This should not be a problem sinc=
e the blocks of my input file are distributed over all nodes.<br>

</div><div><br></div><div>Maybe someone can give me a hint.<br></div><div><=
br>Thanks,<br>Andr=E9 Hacker, TU Berlin<br></div></div>
</blockquote></div><br></div>

--047d7b16013dc5650504e7d92beb--