hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bikas Saha <bi...@hortonworks.com>
Subject RE: Algorithm of distribution Map and Reduce tasks at various topology of a network
Date Tue, 16 Jul 2013 16:50:15 GMT
Can you please share ResourceManager logs, the application id for the job,
the cluster machine configuration.

Thanks
Bikas

-----Original Message-----
From: Jun Ping Du [mailto:jdu@vmware.com]
Sent: Tuesday, July 16, 2013 6:15 AM
To: Костарев А.Ф.
Cc: yarn-dev@hadoop.apache.org
Subject: Re: Algorithm of distribution Map and Reduce tasks at various
topology of a network

Hi Костарев,
I don't think this issue relates to federation. It looks like a1 and a2
doesn't register to RM successfully. It would helpful if you can share
NodeManager's log to see what's wrong with a1 and a2 nodes in registration.

Thanks,

Junping

----- Original Message -----

From: "Костарев А.Ф." <kaf@ics.perm.ru>
To: "Jun Ping Du" <jdu@vmware.com>
Cc: yarn-dev@hadoop.apache.org
Sent: Tuesday, July 16, 2013 6:43:45 PM
Subject: Re: Algorithm of distribution Map and Reduce tasks at various
topology of a network

Hi Jun Ping Du!
Do You use fedaration?

Our cluster works in federation mode. Today we launched tests again. Input
file had size 600 Mb and size of block 6. So, there were 100 map tasks.
And we saw the same result: all map tasks were executed within one
datacenter.

In web-interfaces of HDFS and YARN we saw different information. Page of
HDFS contained information, that there were 6 nodes in cluster, page of YARN
said only ablout 3 nodes.
Screenshots and configuration files are attached.

What settings we need to change to execute tasks on all nodes?


On 07/15/2013 05:52 PM, Jun Ping Du wrote:



Hi Костарев,
I tried to reproduce your case on my 5-nodes setup (with 2 nodes in
dc1/rack1, 1 node in dc1/rack2 and 2 nodes in dc2/rack2) but didn't see
anything unusual. In my test, even with 3 replicas, I saw job with 150 map
tasks distributed across all nodes no matter what datacenter is.
Can you try again with a job with more map tasks as it is pretty random in
scheduling if your job only have 10 map tasks. In your case, it seems b1, b3
and b2 take 3-4 maps away in one heartbeat which is pretty normal case. Let
me know the distribution and version you are using if it still not work with
more tasks.
btw, you can find history log for each job in application page. Isn't it?

Thanks,

Junping




-- 
Консультант 1-й категории
Костарев А.Ф.

Mime
View raw message