hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rammohan ganapavarapu <rammohanga...@gmail.com>
Subject Re: ACCEPTED: waiting for AM container to be allocated, launched and register with RM
Date Mon, 22 Aug 2016 17:27:52 GMT
Thank you all, I have updated my oozie job.properties to use 8030 and now i
am getting below error





    2016-08-22 17:22:02,893 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
NodeManager from node slave03(cmPort: 40511 httpPort: 8042) registered with
capability: <memory:8192, vCores:8>, assigned nodeId slave03:40511
2016-08-22 17:22:02,893 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
slave03:40511 Node Transitioned from NEW to RUNNING
2016-08-22 17:22:02,893 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Added node slave03:40511 clusterResource: <memory:24576, vCores:24>
2016-08-22 17:23:14,258 INFO org.apache.hadoop.ipc.Server: Socket Reader #1
for port 8030: readAndProcess from client 10.16.3.51 threw exception
[org.apache.hadoop.security.AccessControlException: SIMPLE authentication
is not enabled.  Available:[TOKEN]]
org.apache.hadoop.security.AccessControlException: SIMPLE authentication is
not enabled.  Available:[TOKEN]
        at
org.apache.hadoop.ipc.Server$Connection.initializeAuthContext(Server.java:1564)
        at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1520)
        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:771)
        at
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:637)
        at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:608)


So i did enabled simple auth by below config in core-site.xml and restarted
namenode,datanode,rm and nm but still getting same error do i have to do
any thing else to enable simple auth?


    <property>
      <name>hadoop.security.authentication</name>
      <value>simple</value>
    </property>


Ram

On Mon, Aug 22, 2016 at 9:43 AM, Sunil Govind <sunil.govind@gmail.com>
wrote:

> HI Ram
>
> RM logs looks fine and as per config it looks like RM is running on 8030
> itself.
> I am not very sure about the oozie end config which you mentioned. I
> suggest you could check the config end more and debug there.
> Also will let other community folks to pitch in if they have some other
> opinion.
>
> Thanks
> Sunil
>
> On Mon, Aug 22, 2016 at 8:57 PM rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
>
>> any thoughts from the logs and config I have shared?
>>
>> On Aug 21, 2016 8:32 AM, "rammohan ganapavarapu" <rammohanganap@gmail.com>
>> wrote:
>>
>>> so in job.properties what is the jobtracker property, is it RM ip: port
>>> or scheduler port which is 8030, if I use 8030 I am getting unknown
>>> protocol proto buffer error.
>>>
>>> On Aug 21, 2016 7:37 AM, "Sunil Govind" <sunil.govind@gmail.com> wrote:
>>>
>>>> Hi.
>>>>
>>>> It seems its an oozie issue. From conf, RM scheduler is running at port
>>>> 8030.
>>>> But your job.properties is taking 8032. I suggest you could double
>>>> confirm your oozie configuration and see the configurations are intact to
>>>> contact RM. Sharing a link also
>>>> https://discuss.zendesk.com/hc/en-us/articles/203355837-
>>>> How-to-run-a-MapReduce-jar-using-Oozie-workflow
>>>>
>>>> Thanks
>>>> Sunil
>>>>
>>>>
>>>> On Sun, Aug 21, 2016 at 8:41 AM rammohan ganapavarapu <
>>>> rammohanganap@gmail.com> wrote:
>>>>
>>>>> Please find the attached config that i got from yarn ui and  AM,RM
>>>>> logs. I only see that connecting to 0.0.0.0:8030 when i submit job
>>>>> using oozie, but if i submit as yarn jar its working fine as i posted
in my
>>>>> previous posts.
>>>>>
>>>>> Here is my oozie job.properties file, i have a java class that just
>>>>> prints
>>>>>
>>>>> nameNode=hdfs://master01:8020
>>>>> jobTracker=master01:8032
>>>>> workflowName=EchoJavaJob
>>>>> oozie.use.system.libpath=true
>>>>>
>>>>> queueName=default
>>>>> hdfsWorkflowHome=/user/uap/oozieWorkflows
>>>>>
>>>>> workflowPath=${nameNode}${hdfsWorkflowHome}/${workflowName}
>>>>> oozie.wf.application.path=${workflowPath}
>>>>>
>>>>> Please let me know if you guys find any clue why its trying to connect
>>>>> to 0.0.0.:8030.
>>>>>
>>>>> Thanks,
>>>>> Ram
>>>>>
>>>>>
>>>>> On Fri, Aug 19, 2016 at 11:54 PM, Sunil Govind <sunil.govind@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi Ram
>>>>>>
>>>>>> From the console log, as Rohith said, AM is looking for AM at 8030.
>>>>>> So pls confirm the RM port once.
>>>>>> Could you please share AM and RM logs.
>>>>>>
>>>>>> Thanks
>>>>>> Sunil
>>>>>>
>>>>>> On Sat, Aug 20, 2016 at 10:36 AM rammohan ganapavarapu <
>>>>>> rammohanganap@gmail.com> wrote:
>>>>>>
>>>>>>> yes, I did configured.
>>>>>>>
>>>>>>> On Aug 19, 2016 7:22 PM, "Rohith Sharma K S" <
>>>>>>> ksrohithsharma@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> From below discussion and AM logs, I see that AM container
has
>>>>>>>> launched but not able to connect to RM.
>>>>>>>>
>>>>>>>> This looks like your configuration issue. Would you check
your
>>>>>>>> job.xml jar that does *yarn.resourcemanager.scheduler.address
*has
>>>>>>>> been configured?
>>>>>>>>
>>>>>>>> Essentially, this address required by MRAppMaster for connecting
to
>>>>>>>> RM for heartbeats. If you don’t not configure, default
value will be taken
>>>>>>>> i.e 8030.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks & Regards
>>>>>>>> Rohith Sharma K S
>>>>>>>>
>>>>>>>> On Aug 20, 2016, at 7:02 AM, rammohan ganapavarapu <
>>>>>>>> rammohanganap@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Even if  the cluster dont have enough resources it should
connect
>>>>>>>> to "
>>>>>>>>
>>>>>>>> /0.0.0.0:8030" right? it should connect to my <RM_HOST:8030>,
not sure why its trying to connect to 0.0.0.0:8030.
>>>>>>>>
>>>>>>>> I have verified the config and i removed traces of 0.0.0.0
still no luck.
>>>>>>>>
>>>>>>>> org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager
at /0.0.0.0:8030
>>>>>>>>
>>>>>>>> If an one has any clue please share.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Ram
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 19, 2016 at 2:32 PM, rammohan ganapavarapu <
>>>>>>>> rammohanganap@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> When i submit a job using yarn its seems working only
with oozie
>>>>>>>>> its failing i guess, not sure what is missing.
>>>>>>>>>
>>>>>>>>> yarn jar /uap/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar
>>>>>>>>> pi 20 1000
>>>>>>>>> Number of Maps  = 20
>>>>>>>>> Samples per Map = 1000
>>>>>>>>> .
>>>>>>>>> .
>>>>>>>>> .
>>>>>>>>> Job Finished in 19.622 seconds
>>>>>>>>> Estimated value of Pi is 3.14280000000000000000
>>>>>>>>>
>>>>>>>>> Ram
>>>>>>>>>
>>>>>>>>> On Fri, Aug 19, 2016 at 11:46 AM, rammohan ganapavarapu
<
>>>>>>>>> rammohanganap@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Ok, i have used yarn-utils.py to get the correct
values for my
>>>>>>>>>> cluster and update those properties and restarted
RM and NM but still no
>>>>>>>>>> luck not sure what i am missing, any other insights
will help me.
>>>>>>>>>>
>>>>>>>>>> Below are my properties from yarn-site.xml and map-site.xml.
>>>>>>>>>>
>>>>>>>>>> python yarn-utils.py -c 24 -m 63 -d 3 -k False
>>>>>>>>>>  Using cores=24 memory=63GB disks=3 hbase=False
>>>>>>>>>>  Profile: cores=24 memory=63488MB reserved=1GB usableMem=62GB
>>>>>>>>>> disks=3
>>>>>>>>>>  Num Container=6
>>>>>>>>>>  Container Ram=10240MB
>>>>>>>>>>  Used Ram=60GB
>>>>>>>>>>  Unused Ram=1GB
>>>>>>>>>>  yarn.scheduler.minimum-allocation-mb=10240
>>>>>>>>>>  yarn.scheduler.maximum-allocation-mb=61440
>>>>>>>>>>  yarn.nodemanager.resource.memory-mb=61440
>>>>>>>>>>  mapreduce.map.memory.mb=5120
>>>>>>>>>>  mapreduce.map.java.opts=-Xmx4096m
>>>>>>>>>>  mapreduce.reduce.memory.mb=10240
>>>>>>>>>>  mapreduce.reduce.java.opts=-Xmx8192m
>>>>>>>>>>  yarn.app.mapreduce.am.resource.mb=5120
>>>>>>>>>>  yarn.app.mapreduce.am.command-opts=-Xmx4096m
>>>>>>>>>>  mapreduce.task.io.sort.mb=1024
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     <property>
>>>>>>>>>>       <name>mapreduce.map.memory.mb</name>
>>>>>>>>>>       <value>5120</value>
>>>>>>>>>>     </property>
>>>>>>>>>>     <property>
>>>>>>>>>>       <name>mapreduce.map.java.opts</name>
>>>>>>>>>>       <value>-Xmx4096m</value>
>>>>>>>>>>     </property>
>>>>>>>>>>     <property>
>>>>>>>>>>       <name>mapreduce.reduce.memory.mb</name>
>>>>>>>>>>       <value>10240</value>
>>>>>>>>>>     </property>
>>>>>>>>>>     <property>
>>>>>>>>>>       <name>mapreduce.reduce.java.opts</name>
>>>>>>>>>>       <value>-Xmx8192m</value>
>>>>>>>>>>     </property>
>>>>>>>>>>     <property>
>>>>>>>>>>       <name>yarn.app.mapreduce.am.resource.mb</name>
>>>>>>>>>>       <value>5120</value>
>>>>>>>>>>     </property>
>>>>>>>>>>     <property>
>>>>>>>>>>       <name>yarn.app.mapreduce.am.command-opts</name>
>>>>>>>>>>       <value>-Xmx4096m</value>
>>>>>>>>>>     </property>
>>>>>>>>>>     <property>
>>>>>>>>>>       <name>mapreduce.task.io.sort.mb</name>
>>>>>>>>>>       <value>1024</value>
>>>>>>>>>>     </property>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>      <property>
>>>>>>>>>>       <name>yarn.scheduler.minimum-allocation-mb</name>
>>>>>>>>>>       <value>10240</value>
>>>>>>>>>>     </property>
>>>>>>>>>>
>>>>>>>>>>      <property>
>>>>>>>>>>       <name>yarn.scheduler.maximum-allocation-mb</name>
>>>>>>>>>>       <value>61440</value>
>>>>>>>>>>     </property>
>>>>>>>>>>
>>>>>>>>>>      <property>
>>>>>>>>>>       <name>yarn.nodemanager.resource.memory-mb</name>
>>>>>>>>>>       <value>61440</value>
>>>>>>>>>>     </property>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ram
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 18, 2016 at 11:14 PM, tkg_cangkul <
>>>>>>>>>> yuza.rasfar@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> maybe this link can be some reference to tune
up the cluster:
>>>>>>>>>>>
>>>>>>>>>>> http://jason4zhu.blogspot.co.id/2014/10/memory-
>>>>>>>>>>> configuration-in-hadoop.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 19/08/16 11:13, rammohan ganapavarapu wrote:
>>>>>>>>>>>
>>>>>>>>>>> Do you know what properties to tune?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Ram
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 18, 2016 at 9:11 PM, tkg_cangkul
<
>>>>>>>>>>> yuza.rasfar@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> i think that's because you don't have enough
resource.  u can
>>>>>>>>>>>> tune your cluster config to maximize your
resource.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 19/08/16 11:03, rammohan ganapavarapu
wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I dont see any thing odd except this not
sure if i have to
>>>>>>>>>>>> worry about it or not.
>>>>>>>>>>>>
>>>>>>>>>>>> 2016-08-19 03:29:26,621 INFO [main]
>>>>>>>>>>>> org.apache.hadoop.yarn.client.RMProxy: Connecting
to
>>>>>>>>>>>> ResourceManager at /0.0.0.0:8030
>>>>>>>>>>>> 2016-08-19 03:29:27,646 INFO [main]
>>>>>>>>>>>> org.apache.hadoop.ipc.Client: Retrying connect
to server:
>>>>>>>>>>>> 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s);
retry policy is
>>>>>>>>>>>> RetryUpToMaximumCo
>>>>>>>>>>>> untWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
>>>>>>>>>>>> 2016-08-19 03:29:28,647 INFO [main]
>>>>>>>>>>>> org.apache.hadoop.ipc.Client: Retrying connect
to server:
>>>>>>>>>>>> 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s);
retry policy is
>>>>>>>>>>>> RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>>>>>>>>>>>> sleepTime=1000 MILLISECONDS)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> its keep printing this log ..in app container
logs.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 18, 2016 at 8:20 PM, tkg_cangkul
<
>>>>>>>>>>>> yuza.rasfar@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> maybe u can check the logs from port
8088 on your browser.
>>>>>>>>>>>>> that was RM UI. just choose your job
id and then check the logs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 19/08/16 10:14, rammohan ganapavarapu
wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sunil,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks you for your input, below are
my server metrics for RM.
>>>>>>>>>>>>> Also attached RM UI for capacity scheduler
resources. How else i can find?
>>>>>>>>>>>>>
>>>>>>>>>>>>> {
>>>>>>>>>>>>>       "name": "Hadoop:service=ResourceManager,name=
>>>>>>>>>>>>> QueueMetrics,q0=root",
>>>>>>>>>>>>>       "modelerType": "QueueMetrics,q0=root",
>>>>>>>>>>>>>       "tag.Queue": "root",
>>>>>>>>>>>>>       "tag.Context": "yarn",
>>>>>>>>>>>>>       "tag.Hostname": "hadoop001",
>>>>>>>>>>>>>       "running_0": 0,
>>>>>>>>>>>>>       "running_60": 0,
>>>>>>>>>>>>>       "running_300": 0,
>>>>>>>>>>>>>       "running_1440": 0,
>>>>>>>>>>>>>       "AppsSubmitted": 1,
>>>>>>>>>>>>>       "AppsRunning": 0,
>>>>>>>>>>>>>       "AppsPending": 0,
>>>>>>>>>>>>>       "AppsCompleted": 0,
>>>>>>>>>>>>>       "AppsKilled": 0,
>>>>>>>>>>>>>       "AppsFailed": 1,
>>>>>>>>>>>>>       "AllocatedMB": 0,
>>>>>>>>>>>>>       "AllocatedVCores": 0,
>>>>>>>>>>>>>       "AllocatedContainers": 0,
>>>>>>>>>>>>>       "AggregateContainersAllocated":
2,
>>>>>>>>>>>>>       "AggregateContainersReleased":
2,
>>>>>>>>>>>>>       "AvailableMB": 64512,
>>>>>>>>>>>>>       "AvailableVCores": 24,
>>>>>>>>>>>>>       "PendingMB": 0,
>>>>>>>>>>>>>       "PendingVCores": 0,
>>>>>>>>>>>>>       "PendingContainers": 0,
>>>>>>>>>>>>>       "ReservedMB": 0,
>>>>>>>>>>>>>       "ReservedVCores": 0,
>>>>>>>>>>>>>       "ReservedContainers": 0,
>>>>>>>>>>>>>       "ActiveUsers": 0,
>>>>>>>>>>>>>       "ActiveApplications": 0
>>>>>>>>>>>>>     },
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 18, 2016 at 6:49 PM, Sunil
Govind <
>>>>>>>>>>>>> sunil.govind@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It could be because of many of reasons.
Also I am not sure
>>>>>>>>>>>>>> about which scheduler your are using,
pls share more details such as RM log
>>>>>>>>>>>>>> etc.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I could point out few reasons
>>>>>>>>>>>>>>  - Such as "Not enough resource is
cluster" can cause this
>>>>>>>>>>>>>>  - If using Capacity Scheduler, if
queue capacity is maxed
>>>>>>>>>>>>>> out, such case can happen.
>>>>>>>>>>>>>>  - Similarly if max-am-resource-percent
is crossed per queue
>>>>>>>>>>>>>> level, then also AM container may
not be launched.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> you could check RM log to get more
information if AM
>>>>>>>>>>>>>> container is laucnhed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Sunil
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 19, 2016 at 5:37 AM rammohan
ganapavarapu <
>>>>>>>>>>>>>> rammohanganap@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When i submit a MR job, i am
getting this from AM UI but it
>>>>>>>>>>>>>>> never get finished, what am i
missing ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Ram
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>

Mime
View raw message