hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ch huang <justlo...@gmail.com>
Subject Re: default capacity scheduler only one job in running status
Date Wed, 27 Nov 2013 01:14:56 GMT
# yarn application -list
13/11/27 09:13:02 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
Total Applications:2
                Application-Id      Application-Name
User           Queue                   State
Final-State                             Tracking-URL
application_1385514714001_0002  select count(1) from
test_carl_pageview_p(Stage-1)            hive
default                ACCEPTED
UNDEFINED                                      N/A
application_1385514714001_0001  select count(1) from
test_carl_pageview_p(Stage-1)            hive
default                 RUNNING
UNDEFINED                            CHBM222:59768
13/11/27 09:13:03 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.


here is my capacity-scheduler.xml content

<configuration>
  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.3</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>100</value>
    <description>Default queue target capacity.</description>
  </property>


  <property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>0.3</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>
  <property>

<name>yarn.scheduler.capacity.root.default.minimum-user-limit-percent</name>
        <value>25</value>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or
STOPPED.
    </description>
  </property>
  <property>

<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>-1</value>
    <description>
      Number of missed scheduling opportunities after which the
CapacityScheduler
      attempts to schedule rack-local containers.
      Typically this should be set to number of racks in the cluster, this
      feature is disabled by default, set to -1.
    </description>
  </property>
</configuration>





On Wed, Nov 27, 2013 at 8:33 AM, ch huang <justlooks@gmail.com> wrote:

> my config is
>
>   <property>
>     <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
>     <value>0.1</value>
>     <description>
>       Maximum percent of resources in the cluster which can be used to run
>       application masters i.e. controls number of concurrent running
>       applications.
>     </description>
>   </property>
>
>
>  On Tue, Nov 26, 2013 at 5:35 PM, Devaraj K <devaraj@apache.org> wrote:
>
>>  Could you check the below configuration in capacity-scheduler.xml,
>> whether is it causing to run only one AM.
>>
>> <property>
>>     <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
>>     <value>0.1</value>
>>     <description>
>>       Maximum percent of resources in the cluster which can be used to
>> run
>>       application masters i.e. controls number of concurrent running
>>       applications.
>>     </description>
>>   </property>
>>
>>
>> On Tue, Nov 26, 2013 at 2:50 PM, ch huang <justlooks@gmail.com> wrote:
>>
>>> hi,maillist:
>>>             i set the following option in yarn-site.xml ,let yarn
>>> framework to use capacity scheduler,but i submit three job,only one job in
>>> running status,other two stay in accepted status,why ,the default queue
>>> only 50% capacity used,i do not know why?
>>>
>>> <property>
>>>     <name>yarn.resourcemanager.scheduler.class</name>
>>>
>>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>> </property>
>>>
>>
>>
>>
>> --
>>
>>
>> Thanks
>> Devaraj K
>>
>
>

Mime
View raw message