hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil Govind <sunil.gov...@gmail.com>
Subject Re: ACCEPTED: waiting for AM container to be allocated, launched and register with RM
Date Sun, 21 Aug 2016 14:36:49 GMT
Hi.

It seems its an oozie issue. From conf, RM scheduler is running at port
8030.
But your job.properties is taking 8032. I suggest you could double confirm
your oozie configuration and see the configurations are intact to contact
RM. Sharing a link also
https://discuss.zendesk.com/hc/en-us/articles/203355837-How-to-run-a-MapReduce-jar-using-Oozie-workflow

Thanks
Sunil


On Sun, Aug 21, 2016 at 8:41 AM rammohan ganapavarapu <
rammohanganap@gmail.com> wrote:

> Please find the attached config that i got from yarn ui and  AM,RM logs. I
> only see that connecting to 0.0.0.0:8030 when i submit job using oozie,
> but if i submit as yarn jar its working fine as i posted in my previous
> posts.
>
> Here is my oozie job.properties file, i have a java class that just prints
>
> nameNode=hdfs://master01:8020
> jobTracker=master01:8032
> workflowName=EchoJavaJob
> oozie.use.system.libpath=true
>
> queueName=default
> hdfsWorkflowHome=/user/uap/oozieWorkflows
>
> workflowPath=${nameNode}${hdfsWorkflowHome}/${workflowName}
> oozie.wf.application.path=${workflowPath}
>
> Please let me know if you guys find any clue why its trying to connect to
> 0.0.0.:8030.
>
> Thanks,
> Ram
>
>
> On Fri, Aug 19, 2016 at 11:54 PM, Sunil Govind <sunil.govind@gmail.com>
> wrote:
>
>> Hi Ram
>>
>> From the console log, as Rohith said, AM is looking for AM at 8030. So
>> pls confirm the RM port once.
>> Could you please share AM and RM logs.
>>
>> Thanks
>> Sunil
>>
>> On Sat, Aug 20, 2016 at 10:36 AM rammohan ganapavarapu <
>> rammohanganap@gmail.com> wrote:
>>
>>> yes, I did configured.
>>>
>>> On Aug 19, 2016 7:22 PM, "Rohith Sharma K S" <ksrohithsharma@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> From below discussion and AM logs, I see that AM container has launched
>>>> but not able to connect to RM.
>>>>
>>>> This looks like your configuration issue. Would you check your job.xml
>>>> jar that does *yarn.resourcemanager.scheduler.address *has been
>>>> configured?
>>>>
>>>> Essentially, this address required by MRAppMaster for connecting to RM
>>>> for heartbeats. If you don’t not configure, default value will be taken
i.e
>>>> 8030.
>>>>
>>>>
>>>> Thanks & Regards
>>>> Rohith Sharma K S
>>>>
>>>> On Aug 20, 2016, at 7:02 AM, rammohan ganapavarapu <
>>>> rammohanganap@gmail.com> wrote:
>>>>
>>>> Even if  the cluster dont have enough resources it should connect to "
>>>>
>>>> /0.0.0.0:8030" right? it should connect to my <RM_HOST:8030>, not sure
why its trying to connect to 0.0.0.0:8030.
>>>>
>>>> I have verified the config and i removed traces of 0.0.0.0 still no luck.
>>>>
>>>> org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
>>>>
>>>> If an one has any clue please share.
>>>>
>>>> Thanks,
>>>>
>>>> Ram
>>>>
>>>>
>>>>
>>>> On Fri, Aug 19, 2016 at 2:32 PM, rammohan ganapavarapu <
>>>> rammohanganap@gmail.com> wrote:
>>>>
>>>>> When i submit a job using yarn its seems working only with oozie its
>>>>> failing i guess, not sure what is missing.
>>>>>
>>>>> yarn jar
>>>>> /uap/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar
pi
>>>>> 20 1000
>>>>> Number of Maps  = 20
>>>>> Samples per Map = 1000
>>>>> .
>>>>> .
>>>>> .
>>>>> Job Finished in 19.622 seconds
>>>>> Estimated value of Pi is 3.14280000000000000000
>>>>>
>>>>> Ram
>>>>>
>>>>> On Fri, Aug 19, 2016 at 11:46 AM, rammohan ganapavarapu <
>>>>> rammohanganap@gmail.com> wrote:
>>>>>
>>>>>> Ok, i have used yarn-utils.py to get the correct values for my
>>>>>> cluster and update those properties and restarted RM and NM but still
no
>>>>>> luck not sure what i am missing, any other insights will help me.
>>>>>>
>>>>>> Below are my properties from yarn-site.xml and map-site.xml.
>>>>>>
>>>>>> python yarn-utils.py -c 24 -m 63 -d 3 -k False
>>>>>>  Using cores=24 memory=63GB disks=3 hbase=False
>>>>>>  Profile: cores=24 memory=63488MB reserved=1GB usableMem=62GB disks=3
>>>>>>  Num Container=6
>>>>>>  Container Ram=10240MB
>>>>>>  Used Ram=60GB
>>>>>>  Unused Ram=1GB
>>>>>>  yarn.scheduler.minimum-allocation-mb=10240
>>>>>>  yarn.scheduler.maximum-allocation-mb=61440
>>>>>>  yarn.nodemanager.resource.memory-mb=61440
>>>>>>  mapreduce.map.memory.mb=5120
>>>>>>  mapreduce.map.java.opts=-Xmx4096m
>>>>>>  mapreduce.reduce.memory.mb=10240
>>>>>>  mapreduce.reduce.java.opts=-Xmx8192m
>>>>>>  yarn.app.mapreduce.am.resource.mb=5120
>>>>>>  yarn.app.mapreduce.am.command-opts=-Xmx4096m
>>>>>>  mapreduce.task.io.sort.mb=1024
>>>>>>
>>>>>>
>>>>>>     <property>
>>>>>>       <name>mapreduce.map.memory.mb</name>
>>>>>>       <value>5120</value>
>>>>>>     </property>
>>>>>>     <property>
>>>>>>       <name>mapreduce.map.java.opts</name>
>>>>>>       <value>-Xmx4096m</value>
>>>>>>     </property>
>>>>>>     <property>
>>>>>>       <name>mapreduce.reduce.memory.mb</name>
>>>>>>       <value>10240</value>
>>>>>>     </property>
>>>>>>     <property>
>>>>>>       <name>mapreduce.reduce.java.opts</name>
>>>>>>       <value>-Xmx8192m</value>
>>>>>>     </property>
>>>>>>     <property>
>>>>>>       <name>yarn.app.mapreduce.am.resource.mb</name>
>>>>>>       <value>5120</value>
>>>>>>     </property>
>>>>>>     <property>
>>>>>>       <name>yarn.app.mapreduce.am.command-opts</name>
>>>>>>       <value>-Xmx4096m</value>
>>>>>>     </property>
>>>>>>     <property>
>>>>>>       <name>mapreduce.task.io.sort.mb</name>
>>>>>>       <value>1024</value>
>>>>>>     </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>>      <property>
>>>>>>       <name>yarn.scheduler.minimum-allocation-mb</name>
>>>>>>       <value>10240</value>
>>>>>>     </property>
>>>>>>
>>>>>>      <property>
>>>>>>       <name>yarn.scheduler.maximum-allocation-mb</name>
>>>>>>       <value>61440</value>
>>>>>>     </property>
>>>>>>
>>>>>>      <property>
>>>>>>       <name>yarn.nodemanager.resource.memory-mb</name>
>>>>>>       <value>61440</value>
>>>>>>     </property>
>>>>>>
>>>>>>
>>>>>> Ram
>>>>>>
>>>>>> On Thu, Aug 18, 2016 at 11:14 PM, tkg_cangkul <yuza.rasfar@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> maybe this link can be some reference to tune up the cluster:
>>>>>>>
>>>>>>>
>>>>>>> http://jason4zhu.blogspot.co.id/2014/10/memory-configuration-in-hadoop.html
>>>>>>>
>>>>>>>
>>>>>>> On 19/08/16 11:13, rammohan ganapavarapu wrote:
>>>>>>>
>>>>>>> Do you know what properties to tune?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ram
>>>>>>>
>>>>>>> On Thu, Aug 18, 2016 at 9:11 PM, tkg_cangkul <yuza.rasfar@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> i think that's because you don't have enough resource.  u
can tune
>>>>>>>> your cluster config to maximize your resource.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 19/08/16 11:03, rammohan ganapavarapu wrote:
>>>>>>>>
>>>>>>>> I dont see any thing odd except this not sure if i have to
worry
>>>>>>>> about it or not.
>>>>>>>>
>>>>>>>> 2016-08-19 03:29:26,621 INFO [main]
>>>>>>>> org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager
at /
>>>>>>>> 0.0.0.0:8030
>>>>>>>> 2016-08-19 03:29:27,646 INFO [main] org.apache.hadoop.ipc.Client:
>>>>>>>> Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already
tried 0
>>>>>>>> time(s); retry policy is RetryUpToMaximumCo
>>>>>>>> untWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
>>>>>>>> 2016-08-19 03:29:28,647 INFO [main] org.apache.hadoop.ipc.Client:
>>>>>>>> Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already
tried 1
>>>>>>>> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>>>>>>>> sleepTime=1000 MILLISECONDS)
>>>>>>>>
>>>>>>>>
>>>>>>>> its keep printing this log ..in app container logs.
>>>>>>>>
>>>>>>>> On Thu, Aug 18, 2016 at 8:20 PM, tkg_cangkul <yuza.rasfar@gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> maybe u can check the logs from port 8088 on your browser.
that
>>>>>>>>> was RM UI. just choose your job id and then check the
logs.
>>>>>>>>>
>>>>>>>>> On 19/08/16 10:14, rammohan ganapavarapu wrote:
>>>>>>>>>
>>>>>>>>> Sunil,
>>>>>>>>>
>>>>>>>>> Thanks you for your input, below are my server metrics
for RM.
>>>>>>>>> Also attached RM UI for capacity scheduler resources.
How else i can find?
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>>       "name":
>>>>>>>>> "Hadoop:service=ResourceManager,name=QueueMetrics,q0=root",
>>>>>>>>>       "modelerType": "QueueMetrics,q0=root",
>>>>>>>>>       "tag.Queue": "root",
>>>>>>>>>       "tag.Context": "yarn",
>>>>>>>>>       "tag.Hostname": "hadoop001",
>>>>>>>>>       "running_0": 0,
>>>>>>>>>       "running_60": 0,
>>>>>>>>>       "running_300": 0,
>>>>>>>>>       "running_1440": 0,
>>>>>>>>>       "AppsSubmitted": 1,
>>>>>>>>>       "AppsRunning": 0,
>>>>>>>>>       "AppsPending": 0,
>>>>>>>>>       "AppsCompleted": 0,
>>>>>>>>>       "AppsKilled": 0,
>>>>>>>>>       "AppsFailed": 1,
>>>>>>>>>       "AllocatedMB": 0,
>>>>>>>>>       "AllocatedVCores": 0,
>>>>>>>>>       "AllocatedContainers": 0,
>>>>>>>>>       "AggregateContainersAllocated": 2,
>>>>>>>>>       "AggregateContainersReleased": 2,
>>>>>>>>>       "AvailableMB": 64512,
>>>>>>>>>       "AvailableVCores": 24,
>>>>>>>>>       "PendingMB": 0,
>>>>>>>>>       "PendingVCores": 0,
>>>>>>>>>       "PendingContainers": 0,
>>>>>>>>>       "ReservedMB": 0,
>>>>>>>>>       "ReservedVCores": 0,
>>>>>>>>>       "ReservedContainers": 0,
>>>>>>>>>       "ActiveUsers": 0,
>>>>>>>>>       "ActiveApplications": 0
>>>>>>>>>     },
>>>>>>>>>
>>>>>>>>> On Thu, Aug 18, 2016 at 6:49 PM, Sunil Govind <
>>>>>>>>> sunil.govind@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> It could be because of many of reasons. Also I am
not sure about
>>>>>>>>>> which scheduler your are using, pls share more details
such as RM log etc.
>>>>>>>>>>
>>>>>>>>>> I could point out few reasons
>>>>>>>>>>  - Such as "Not enough resource is cluster" can cause
this
>>>>>>>>>>  - If using Capacity Scheduler, if queue capacity
is maxed out,
>>>>>>>>>> such case can happen.
>>>>>>>>>>  - Similarly if max-am-resource-percent is crossed
per queue
>>>>>>>>>> level, then also AM container may not be launched.
>>>>>>>>>>
>>>>>>>>>> you could check RM log to get more information if
AM container is
>>>>>>>>>> laucnhed.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Sunil
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 19, 2016 at 5:37 AM rammohan ganapavarapu
<
>>>>>>>>>> rammohanganap@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> When i submit a MR job, i am getting this from
AM UI but it
>>>>>>>>>>> never get finished, what am i missing ?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Ram
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>>>>>>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>

Mime
View raw message