hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <ksrohithsha...@gmail.com>
Subject Re: ACCEPTED: waiting for AM container to be allocated, launched and register with RM
Date Sat, 20 Aug 2016 02:22:27 GMT
Hi

From below discussion and AM logs, I see that AM container has launched but not able to connect
to RM.

This looks like your configuration issue. Would you check your job.xml jar that does yarn.resourcemanager.scheduler.address
has been configured? 

Essentially, this address required by MRAppMaster for connecting to RM for heartbeats. If
you don’t not configure, default value will be taken i.e 8030.


Thanks & Regards
Rohith Sharma K S

> On Aug 20, 2016, at 7:02 AM, rammohan ganapavarapu <rammohanganap@gmail.com> wrote:
> 
> Even if  the cluster dont have enough resources it should connect to "
> /0.0.0.0:8030 <http://0.0.0.0:8030/>" right? it should connect to my <RM_HOST:8030>,
not sure why its trying to connect to 0.0.0.0:8030 <http://0.0.0.0:8030/>.
> I have verified the config and i removed traces of 0.0.0.0 still no luck.
> org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
<http://0.0.0.0:8030/>
> 
> If an one has any clue please share.
> 
> Thanks,
> Ram
> 
> 
> On Fri, Aug 19, 2016 at 2:32 PM, rammohan ganapavarapu <rammohanganap@gmail.com <mailto:rammohanganap@gmail.com>>
wrote:
> When i submit a job using yarn its seems working only with oozie its failing i guess,
not sure what is missing.
> 
> yarn jar /uap/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 20
1000
> Number of Maps  = 20
> Samples per Map = 1000
> .
> .
> .
> Job Finished in 19.622 seconds
> Estimated value of Pi is 3.14280000000000000000
> 
> Ram
> 
> On Fri, Aug 19, 2016 at 11:46 AM, rammohan ganapavarapu <rammohanganap@gmail.com <mailto:rammohanganap@gmail.com>>
wrote:
> Ok, i have used yarn-utils.py to get the correct values for my cluster and update those
properties and restarted RM and NM but still no luck not sure what i am missing, any other
insights will help me.
> 
> Below are my properties from yarn-site.xml and map-site.xml.
> 
> python yarn-utils.py -c 24 -m 63 -d 3 -k False
>  Using cores=24 memory=63GB disks=3 hbase=False
>  Profile: cores=24 memory=63488MB reserved=1GB usableMem=62GB disks=3
>  Num Container=6
>  Container Ram=10240MB
>  Used Ram=60GB
>  Unused Ram=1GB
>  yarn.scheduler.minimum-allocation-mb=10240
>  yarn.scheduler.maximum-allocation-mb=61440
>  yarn.nodemanager.resource.memory-mb=61440
>  mapreduce.map.memory.mb=5120
>  mapreduce.map.java.opts=-Xmx4096m
>  mapreduce.reduce.memory.mb=10240
>  mapreduce.reduce.java.opts=-Xmx8192m
>  yarn.app.mapreduce.am <http://yarn.app.mapreduce.am/>.resource.mb=5120
>  yarn.app.mapreduce.am <http://yarn.app.mapreduce.am/>.command-opts=-Xmx4096m
>  mapreduce.task.io.sort.mb=1024
> 
> 
>     <property>
>       <name>mapreduce.map.memory.mb</name>
>       <value>5120</value>
>     </property>
>     <property>
>       <name>mapreduce.map.java.opts</name>
>       <value>-Xmx4096m</value>
>     </property>
>     <property>
>       <name>mapreduce.reduce.memory.mb</name>
>       <value>10240</value>
>     </property>
>     <property>
>       <name>mapreduce.reduce.java.opts</name>
>       <value>-Xmx8192m</value>
>     </property>
>     <property>
>       <name>yarn.app.mapreduce.am <http://yarn.app.mapreduce.am/>.resource.mb</name>
>       <value>5120</value>
>     </property>
>     <property>
>       <name>yarn.app.mapreduce.am <http://yarn.app.mapreduce.am/>.command-opts</name>
>       <value>-Xmx4096m</value>
>     </property>
>     <property>
>       <name>mapreduce.task.io.sort.mb</name>
>       <value>1024</value>
>     </property>
> 
> 
> 
>      <property>
>       <name>yarn.scheduler.minimum-allocation-mb</name>
>       <value>10240</value>
>     </property>
> 
>      <property>
>       <name>yarn.scheduler.maximum-allocation-mb</name>
>       <value>61440</value>
>     </property>
> 
>      <property>
>       <name>yarn.nodemanager.resource.memory-mb</name>
>       <value>61440</value>
>     </property>
> 
> 
> Ram
> 
> On Thu, Aug 18, 2016 at 11:14 PM, tkg_cangkul <yuza.rasfar@gmail.com <mailto:yuza.rasfar@gmail.com>>
wrote:
> maybe this link can be some reference to tune up the cluster:
> 
> http://jason4zhu.blogspot.co.id/2014/10/memory-configuration-in-hadoop.html <http://jason4zhu.blogspot.co.id/2014/10/memory-configuration-in-hadoop.html>
> 
> 
> On 19/08/16 11:13, rammohan ganapavarapu wrote:
>> Do you know what properties to tune?
>> 
>> Thanks,
>> Ram
>> 
>> On Thu, Aug 18, 2016 at 9:11 PM, tkg_cangkul <yuza.rasfar@gmail.com <mailto:yuza.rasfar@gmail.com>>
wrote:
>> i think that's because you don't have enough resource.  u can tune your cluster config
to maximize your resource.
>> 
>> 
>> On 19/08/16 11:03, rammohan ganapavarapu wrote:
>>> I dont see any thing odd except this not sure if i have to worry about it or
not.
>>> 
>>> 2016-08-19 03:29:26,621 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting
to ResourceManager at /0.0.0.0:8030 <http://0.0.0.0:8030/>
>>> 2016-08-19 03:29:27,646 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect
to server: 0.0.0.0/0.0.0.0:8030 <http://0.0.0.0/0.0.0.0:8030>. Already tried 0 time(s);
retry policy is RetryUpToMaximumCo
>>> untWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
>>> 2016-08-19 03:29:28,647 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect
to server: 0.0.0.0/0.0.0.0:8030 <http://0.0.0.0/0.0.0.0:8030>. Already tried 1 time(s);
retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
>>> 
>>> 
>>> its keep printing this log ..in app container logs.
>>> 
>>> On Thu, Aug 18, 2016 at 8:20 PM, tkg_cangkul <yuza.rasfar@gmail.com <mailto:yuza.rasfar@gmail.com>>
wrote:
>>> maybe u can check the logs from port 8088 on your browser. that was RM UI. just
choose your job id and then check the logs. 
>>>  
>>> On 19/08/16 10:14, rammohan ganapavarapu wrote:
>>>> Sunil,
>>>> 
>>>> Thanks you for your input, below are my server metrics for RM. Also attached
RM UI for capacity scheduler resources. How else i can find?
>>>> 
>>>> {
>>>>       "name": "Hadoop:service=ResourceManager,name=QueueMetrics,q0=root",
>>>>       "modelerType": "QueueMetrics,q0=root",
>>>>       "tag.Queue": "root",
>>>>       "tag.Context": "yarn",
>>>>       "tag.Hostname": "hadoop001",
>>>>       "running_0": 0,
>>>>       "running_60": 0,
>>>>       "running_300": 0,
>>>>       "running_1440": 0,
>>>>       "AppsSubmitted": 1,
>>>>       "AppsRunning": 0,
>>>>       "AppsPending": 0,
>>>>       "AppsCompleted": 0,
>>>>       "AppsKilled": 0,
>>>>       "AppsFailed": 1,
>>>>       "AllocatedMB": 0,
>>>>       "AllocatedVCores": 0,
>>>>       "AllocatedContainers": 0,
>>>>       "AggregateContainersAllocated": 2,
>>>>       "AggregateContainersReleased": 2,
>>>>       "AvailableMB": 64512,
>>>>       "AvailableVCores": 24,
>>>>       "PendingMB": 0,
>>>>       "PendingVCores": 0,
>>>>       "PendingContainers": 0,
>>>>       "ReservedMB": 0,
>>>>       "ReservedVCores": 0,
>>>>       "ReservedContainers": 0,
>>>>       "ActiveUsers": 0,
>>>>       "ActiveApplications": 0
>>>>     },
>>>> 
>>>> On Thu, Aug 18, 2016 at 6:49 PM, Sunil Govind <sunil.govind@gmail.com
<mailto:sunil.govind@gmail.com>> wrote:
>>>> Hi
>>>> 
>>>> It could be because of many of reasons. Also I am not sure about which scheduler
your are using, pls share more details such as RM log etc.
>>>> 
>>>> I could point out few reasons
>>>>  - Such as "Not enough resource is cluster" can cause this
>>>>  - If using Capacity Scheduler, if queue capacity is maxed out, such case
can happen.
>>>>  - Similarly if max-am-resource-percent is crossed per queue level, then
also AM container may not be launched.
>>>> 
>>>> you could check RM log to get more information if AM container is laucnhed.
>>>> 
>>>> Thanks
>>>> Sunil
>>>> 
>>>> On Fri, Aug 19, 2016 at 5:37 AM rammohan ganapavarapu <rammohanganap@gmail.com
<mailto:rammohanganap@gmail.com>> wrote:
>>>> Hi,
>>>> 
>>>> When i submit a MR job, i am getting this from AM UI but it never get finished,
what am i missing ?
>>>> 
>>>> Thanks,
>>>> Ram
>>>> 
>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org <mailto:user-unsubscribe@hadoop.apache.org>
>>>> For additional commands, e-mail: user-help@hadoop.apache.org <mailto:user-help@hadoop.apache.org>
>>> 
>> 
>> 
> 
> 
> 
> 


Mime
View raw message