hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phillip Wu <phillip...@unsw.edu.au>
Subject RE: maximum-am-resource-percent is insufficient to start a single application
Date Wed, 15 Jun 2016 07:27:42 GMT
Thanks for your help.

I’ve run the http and this is it:
Apps Submitted               Apps Pending    Apps Running    Apps Completed              Containers
Running        Memory Used                Memory Total   Memory Reserved          VCores Used
     VCores Total      VCores Reserved             Active Nodes                Decommissioned
Nodes               Lost Nodes         Unhealthy Nodes            Rebooted Nodes
1              1              0              0              0              0 B          0
B          0 B          0              0              0              0              0    
         0              0              0
ID User Name Application Type Queue StartTime FinishTime State FinalStatus Progress Tracking
UI Blacklisted Nodes
application_1465974687991_0001             hduser  Insert into CATEGORIES Va...'beverages.gif')(Stage-1)
   MAPREDUCE                default Wed, 15 Jun 2016 07:13:31 GMT                N/A     
  ACCEPTED           UNDEFINED
ApplicationMaster           0

I’ve attached a jpeg of the http & resource manager log.

The resource log looks like:
2016-06-15 07:13:30,152 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService:
Allocated new applicationId: 1
2016-06-15 07:13:31,906 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
Storing application with id application_1465974687991_0001
2016-06-15 07:13:31,912 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1465974687991_0001 State change from NEW to NEW_SAVING
2016-06-15 07:13:31,917 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService:
Application with id 1 submitted by user hduser
2016-06-15 07:13:31,923 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
Storing info for app: application_1465974687991_0001
2016-06-15 07:13:31,925 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
USER=hduser   IP=127.0.0.1    OPERATION=Submit Application Request    TARGET=ClientRMService
 RESULT=SUCCESS  APPID=application_1465974687991_0001
2016-06-15 07:13:31,928 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1465974687991_0001 State change from NEW_SAVING to SUBMITTED
2016-06-15 07:13:31,930 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Application added - appId: application_1465974687991_0001 user: hduser leaf-queue of parent:
root #applications: 1
2016-06-15 07:13:31,931 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Accepted application application_1465974687991_0001 from user: hduser, in queue: default
2016-06-15 07:13:31,947 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1465974687991_0001 State change from SUBMITTED to ACCEPTED
2016-06-15 07:13:32,001 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Registering app attempt : appattempt_1465974687991_0001_000001
2016-06-15 07:13:32,002 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1465974687991_0001_000001 State change from NEW to SUBMITTED
2016-06-15 07:13:32,034 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue, it is
likely set too low. skipping enforcement to allow at least one application to start
2016-06-15 07:13:32,034 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue for user,
it is likely set too low. skipping enforcement to allow at least one application to start
2016-06-15 07:13:32,035 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application application_1465974687991_0001 from user: hduser activated in queue: default
2016-06-15 07:13:32,035 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application added - appId: application_1465974687991_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@6593b169,
leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications:
0 #queue-active-applications: 1
2016-06-15 07:13:32,035 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Added Application Attempt appattempt_1465974687991_0001_000001 to scheduler from user hduser
in queue default
2016-06-15 07:13:32,036 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1465974687991_0001_000001 State change from SUBMITTED to SCHEDULED
2016-06-15 07:21:27,980 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
Release request cache is cleaned up

Phillip

From: Sunil Govind [mailto:sunil.govind@gmail.com]
Sent: Wednesday, 15 June 2016 5:11 PM
To: Varun saxena; Phillip Wu; user@hadoop.apache.org
Subject: Re: maximum-am-resource-percent is insufficient to start a single application

Adding to what Varun has said, Resource Manager log will be of help here to confirm same.

The code snippet which you have mentioned is correct. But it also has a check that if the
number of active application is less than 1, this check wont be performed. And it seems you
have only one application.

- Sunil



On Wed, Jun 15, 2016 at 12:27 PM Varun saxena <varun.saxena@huawei.com<mailto:varun.saxena@huawei.com>>
wrote:
Can you open the Resource Manager(RM) UI and share screenshot of main RM page. We can check
cluster resources there. Most probably cluster does not have enough resources.
How much memory and VCores does your AM need ?
RM UI can be accessed at http://localhost:8088/

- Varun Saxena.

From: Phillip Wu [mailto:phillip.wu@unsw.edu.au<mailto:phillip.wu@unsw.edu.au>]
Sent: 15 June 2016 14:42
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Cc: Sunil Govind
Subject: RE: maximum-am-resource-percent is insufficient to start a single application

Sunil,

Thanks for your email.


1.       I don’t think anything on the cluster is being used – see below

I’m not sure how to get my “total cluster resource size” – please advise how to get
this?
After doing the hive insert I get this:
hduser@ip-10-118-112-182:/$ hadoop queue -info default -showJobs
16/06/10 02:24:49 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8050<http://127.0.0.1:8050>
======================
Queue Name : default
Queue State : running
Scheduling Info : Capacity: 100.0, MaximumCapacity: 100.0, CurrentCapacity: 0.0
Total jobs:1
                  JobId      State           StartTime      UserName           Queue     
Priority       UsedContainers  RsvdContainers  UsedMem         RsvdMem         NeededMem 
       AM info
job_1465523894946_0001       PREP       1465524072194        hduser         default      
 NORMAL                    0               0       0M              0M                0M  
   http://localhost:8088/proxy/application_1465523894946_0001/

hduser@ip-10-118-112-182:/$ mapred job -status  job_1465523894946_0001
Job: job_1465523894946_0001
Job File: /tmp/hadoop-yarn/staging/hduser/.staging/job_1465523894946_0001/job.xml
Job Tracking URL : http://localhost:8088/proxy/application_1465523894946_0001/
Uber job : false
Number of maps: 0
Number of reduces: 0
map() completion: 0.0
reduce() completion: 0.0
Job state: PREP
retired: false
reason for failure:
Counters: 0

2.       There are no other applications except I’m running zookeeper

3.       There is only one user

For your assistance this seems to be the code generating the error message[…yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java]:
if (!Resources.lessThanOrEqual(
          resourceCalculator, lastClusterResource, userAmIfStarted,
          userAMLimit)) {
        if (getNumActiveApplications() < 1) {
          LOG.warn("maximum-am-resource-percent is insufficient to start a" +
            " single application in queue for user, it is likely set too low." +
            " skipping enforcement to allow at least one application to start");
        } else {
          LOG.info("not starting application as amIfStarted exceeds " +
            "userAmLimit");
          continue;
        }
      }

Any ideas?

Phillip
From: Sunil Govind [mailto:sunil.govind@gmail.com]
Sent: Wednesday, 15 June 2016 4:24 PM
To: Phillip Wu; user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: maximum-am-resource-percent is insufficient to start a single application

Hi Philip

Higher maximum-am-resource-percent value (0~1) will help to allocate more resource for your
ApplicationMaster container of a yarn application (MR Jobs here), but also depend on the capacity
configured for the queue. You have mentioned that there is only default queue here, so that
wont be a problem. Few questions:
    - How much is your total cluster resource size and how much of cluster resource is used
now ?
    - Is there any other application were running in cluster and whether it was taking full
cluster resource.? This is a possibility since you now gave whole queue's capacity for AM
containers.
    - Do you have multiple users in your cluster who runs applications other that this hive
job? If so, yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent will have
impact on AM resource usage limit. I think you can double check this.


- Sunil

On Wed, Jun 15, 2016 at 8:47 AM Phillip Wu <phillip.wu@unsw.edu.au<mailto:phillip.wu@unsw.edu.au>>
wrote:
Hi,

I'm new to Hadoop and Hive.

I'm using Hadoop 2.6.4 (binary I got from internet) & Hive 2.0.1 (binary I got from internet).
I can create a database and table in hive.

However when I try to insert a record into a previously created table I get:
"org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent
is insufficient to start a single application in queue"

yarn-site.xml
<property>
      <name>yarn.resourcemanager.scheduler.class</name>
      <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

capacity-scheduler.xml
<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>1.0</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

According to the documentation this means I have allocated 100% to my one and only default
scheduler queue.
[https://hadoop.apache.org/docs/r2.6.4/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html]
"yarn.scheduler.capacity.maximum-am-resource-percent / yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent
Maximum percent of resources in the cluster which can be used to run application masters -
controls number of concurrent active applications.
Limits on each queue are directly proportional to their queue capacities and user limits.
Specified as a float - ie 0.5 = 50%. Default is 10%. This can be set for all queues with yarn.scheduler.capacity.maximum-am-resource-percent
and can also be overridden on a per queue basis by setting
yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent"

Can someone tell me how to fix this?
Mime
View raw message