hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rammohan ganapavarapu <rammohanga...@gmail.com>
Subject Re: ACCEPTED: waiting for AM container to be allocated, launched and register with RM
Date Sun, 21 Aug 2016 03:10:44 GMT
Please find the attached config that i got from yarn ui and  AM,RM logs. I
only see that connecting to 0.0.0.0:8030 when i submit job using oozie, but
if i submit as yarn jar its working fine as i posted in my previous posts.

Here is my oozie job.properties file, i have a java class that just prints

nameNode=hdfs://master01:8020
jobTracker=master01:8032
workflowName=EchoJavaJob
oozie.use.system.libpath=true

queueName=default
hdfsWorkflowHome=/user/uap/oozieWorkflows

workflowPath=${nameNode}${hdfsWorkflowHome}/${workflowName}
oozie.wf.application.path=${workflowPath}

Please let me know if you guys find any clue why its trying to connect to
0.0.0.:8030.

Thanks,
Ram


On Fri, Aug 19, 2016 at 11:54 PM, Sunil Govind <sunil.govind@gmail.com>
wrote:

> Hi Ram
>
> From the console log, as Rohith said, AM is looking for AM at 8030. So pls
> confirm the RM port once.
> Could you please share AM and RM logs.
>
> Thanks
> Sunil
>
> On Sat, Aug 20, 2016 at 10:36 AM rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
>
>> yes, I did configured.
>>
>> On Aug 19, 2016 7:22 PM, "Rohith Sharma K S" <ksrohithsharma@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> From below discussion and AM logs, I see that AM container has launched
>>> but not able to connect to RM.
>>>
>>> This looks like your configuration issue. Would you check your job.xml
>>> jar that does *yarn.resourcemanager.scheduler.address *has been
>>> configured?
>>>
>>> Essentially, this address required by MRAppMaster for connecting to RM
>>> for heartbeats. If you don’t not configure, default value will be taken i.e
>>> 8030.
>>>
>>>
>>> Thanks & Regards
>>> Rohith Sharma K S
>>>
>>> On Aug 20, 2016, at 7:02 AM, rammohan ganapavarapu <
>>> rammohanganap@gmail.com> wrote:
>>>
>>> Even if  the cluster dont have enough resources it should connect to "
>>>
>>> /0.0.0.0:8030" right? it should connect to my <RM_HOST:8030>, not sure
why its trying to connect to 0.0.0.0:8030.
>>>
>>> I have verified the config and i removed traces of 0.0.0.0 still no luck.
>>>
>>> org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
>>>
>>> If an one has any clue please share.
>>>
>>> Thanks,
>>>
>>> Ram
>>>
>>>
>>>
>>> On Fri, Aug 19, 2016 at 2:32 PM, rammohan ganapavarapu <
>>> rammohanganap@gmail.com> wrote:
>>>
>>>> When i submit a job using yarn its seems working only with oozie its
>>>> failing i guess, not sure what is missing.
>>>>
>>>> yarn jar /uap/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar
>>>> pi 20 1000
>>>> Number of Maps  = 20
>>>> Samples per Map = 1000
>>>> .
>>>> .
>>>> .
>>>> Job Finished in 19.622 seconds
>>>> Estimated value of Pi is 3.14280000000000000000
>>>>
>>>> Ram
>>>>
>>>> On Fri, Aug 19, 2016 at 11:46 AM, rammohan ganapavarapu <
>>>> rammohanganap@gmail.com> wrote:
>>>>
>>>>> Ok, i have used yarn-utils.py to get the correct values for my cluster
>>>>> and update those properties and restarted RM and NM but still no luck
not
>>>>> sure what i am missing, any other insights will help me.
>>>>>
>>>>> Below are my properties from yarn-site.xml and map-site.xml.
>>>>>
>>>>> python yarn-utils.py -c 24 -m 63 -d 3 -k False
>>>>>  Using cores=24 memory=63GB disks=3 hbase=False
>>>>>  Profile: cores=24 memory=63488MB reserved=1GB usableMem=62GB disks=3
>>>>>  Num Container=6
>>>>>  Container Ram=10240MB
>>>>>  Used Ram=60GB
>>>>>  Unused Ram=1GB
>>>>>  yarn.scheduler.minimum-allocation-mb=10240
>>>>>  yarn.scheduler.maximum-allocation-mb=61440
>>>>>  yarn.nodemanager.resource.memory-mb=61440
>>>>>  mapreduce.map.memory.mb=5120
>>>>>  mapreduce.map.java.opts=-Xmx4096m
>>>>>  mapreduce.reduce.memory.mb=10240
>>>>>  mapreduce.reduce.java.opts=-Xmx8192m
>>>>>  yarn.app.mapreduce.am.resource.mb=5120
>>>>>  yarn.app.mapreduce.am.command-opts=-Xmx4096m
>>>>>  mapreduce.task.io.sort.mb=1024
>>>>>
>>>>>
>>>>>     <property>
>>>>>       <name>mapreduce.map.memory.mb</name>
>>>>>       <value>5120</value>
>>>>>     </property>
>>>>>     <property>
>>>>>       <name>mapreduce.map.java.opts</name>
>>>>>       <value>-Xmx4096m</value>
>>>>>     </property>
>>>>>     <property>
>>>>>       <name>mapreduce.reduce.memory.mb</name>
>>>>>       <value>10240</value>
>>>>>     </property>
>>>>>     <property>
>>>>>       <name>mapreduce.reduce.java.opts</name>
>>>>>       <value>-Xmx8192m</value>
>>>>>     </property>
>>>>>     <property>
>>>>>       <name>yarn.app.mapreduce.am.resource.mb</name>
>>>>>       <value>5120</value>
>>>>>     </property>
>>>>>     <property>
>>>>>       <name>yarn.app.mapreduce.am.command-opts</name>
>>>>>       <value>-Xmx4096m</value>
>>>>>     </property>
>>>>>     <property>
>>>>>       <name>mapreduce.task.io.sort.mb</name>
>>>>>       <value>1024</value>
>>>>>     </property>
>>>>>
>>>>>
>>>>>
>>>>>      <property>
>>>>>       <name>yarn.scheduler.minimum-allocation-mb</name>
>>>>>       <value>10240</value>
>>>>>     </property>
>>>>>
>>>>>      <property>
>>>>>       <name>yarn.scheduler.maximum-allocation-mb</name>
>>>>>       <value>61440</value>
>>>>>     </property>
>>>>>
>>>>>      <property>
>>>>>       <name>yarn.nodemanager.resource.memory-mb</name>
>>>>>       <value>61440</value>
>>>>>     </property>
>>>>>
>>>>>
>>>>> Ram
>>>>>
>>>>> On Thu, Aug 18, 2016 at 11:14 PM, tkg_cangkul <yuza.rasfar@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> maybe this link can be some reference to tune up the cluster:
>>>>>>
>>>>>> http://jason4zhu.blogspot.co.id/2014/10/memory-
>>>>>> configuration-in-hadoop.html
>>>>>>
>>>>>>
>>>>>> On 19/08/16 11:13, rammohan ganapavarapu wrote:
>>>>>>
>>>>>> Do you know what properties to tune?
>>>>>>
>>>>>> Thanks,
>>>>>> Ram
>>>>>>
>>>>>> On Thu, Aug 18, 2016 at 9:11 PM, tkg_cangkul <yuza.rasfar@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> i think that's because you don't have enough resource.  u can
tune
>>>>>>> your cluster config to maximize your resource.
>>>>>>>
>>>>>>>
>>>>>>> On 19/08/16 11:03, rammohan ganapavarapu wrote:
>>>>>>>
>>>>>>> I dont see any thing odd except this not sure if i have to worry
>>>>>>> about it or not.
>>>>>>>
>>>>>>> 2016-08-19 03:29:26,621 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
>>>>>>> Connecting to ResourceManager at /0.0.0.0:8030
>>>>>>> 2016-08-19 03:29:27,646 INFO [main] org.apache.hadoop.ipc.Client:
>>>>>>> Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried
0
>>>>>>> time(s); retry policy is RetryUpToMaximumCo
>>>>>>> untWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
>>>>>>> 2016-08-19 03:29:28,647 INFO [main] org.apache.hadoop.ipc.Client:
>>>>>>> Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried
1
>>>>>>> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>>>>>>> sleepTime=1000 MILLISECONDS)
>>>>>>>
>>>>>>>
>>>>>>> its keep printing this log ..in app container logs.
>>>>>>>
>>>>>>> On Thu, Aug 18, 2016 at 8:20 PM, tkg_cangkul <yuza.rasfar@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> maybe u can check the logs from port 8088 on your browser.
that was
>>>>>>>> RM UI. just choose your job id and then check the logs.
>>>>>>>>
>>>>>>>> On 19/08/16 10:14, rammohan ganapavarapu wrote:
>>>>>>>>
>>>>>>>> Sunil,
>>>>>>>>
>>>>>>>> Thanks you for your input, below are my server metrics for
RM. Also
>>>>>>>> attached RM UI for capacity scheduler resources. How else
i can find?
>>>>>>>>
>>>>>>>> {
>>>>>>>>       "name": "Hadoop:service=ResourceManager,name=
>>>>>>>> QueueMetrics,q0=root",
>>>>>>>>       "modelerType": "QueueMetrics,q0=root",
>>>>>>>>       "tag.Queue": "root",
>>>>>>>>       "tag.Context": "yarn",
>>>>>>>>       "tag.Hostname": "hadoop001",
>>>>>>>>       "running_0": 0,
>>>>>>>>       "running_60": 0,
>>>>>>>>       "running_300": 0,
>>>>>>>>       "running_1440": 0,
>>>>>>>>       "AppsSubmitted": 1,
>>>>>>>>       "AppsRunning": 0,
>>>>>>>>       "AppsPending": 0,
>>>>>>>>       "AppsCompleted": 0,
>>>>>>>>       "AppsKilled": 0,
>>>>>>>>       "AppsFailed": 1,
>>>>>>>>       "AllocatedMB": 0,
>>>>>>>>       "AllocatedVCores": 0,
>>>>>>>>       "AllocatedContainers": 0,
>>>>>>>>       "AggregateContainersAllocated": 2,
>>>>>>>>       "AggregateContainersReleased": 2,
>>>>>>>>       "AvailableMB": 64512,
>>>>>>>>       "AvailableVCores": 24,
>>>>>>>>       "PendingMB": 0,
>>>>>>>>       "PendingVCores": 0,
>>>>>>>>       "PendingContainers": 0,
>>>>>>>>       "ReservedMB": 0,
>>>>>>>>       "ReservedVCores": 0,
>>>>>>>>       "ReservedContainers": 0,
>>>>>>>>       "ActiveUsers": 0,
>>>>>>>>       "ActiveApplications": 0
>>>>>>>>     },
>>>>>>>>
>>>>>>>> On Thu, Aug 18, 2016 at 6:49 PM, Sunil Govind <
>>>>>>>> sunil.govind@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> It could be because of many of reasons. Also I am not
sure about
>>>>>>>>> which scheduler your are using, pls share more details
such as RM log etc.
>>>>>>>>>
>>>>>>>>> I could point out few reasons
>>>>>>>>>  - Such as "Not enough resource is cluster" can cause
this
>>>>>>>>>  - If using Capacity Scheduler, if queue capacity is
maxed out,
>>>>>>>>> such case can happen.
>>>>>>>>>  - Similarly if max-am-resource-percent is crossed per
queue
>>>>>>>>> level, then also AM container may not be launched.
>>>>>>>>>
>>>>>>>>> you could check RM log to get more information if AM
container is
>>>>>>>>> laucnhed.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Sunil
>>>>>>>>>
>>>>>>>>> On Fri, Aug 19, 2016 at 5:37 AM rammohan ganapavarapu
<
>>>>>>>>> rammohanganap@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> When i submit a MR job, i am getting this from AM
UI but it never
>>>>>>>>>> get finished, what am i missing ?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ram
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>>>>>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>

Mime
View raw message