hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Riccomini (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
Date Wed, 12 Jun 2013 18:38:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681486#comment-13681486
] 

Chris Riccomini commented on YARN-600:
--------------------------------------

Hey Sandy,

I just applied your patch to my local YARN, and can verify that it appears to be working.

{noformat}
$ cat container_1371061837111_0001_01_000002/cpu.shares 
1024
$ cat container_1371061837111_0002_01_000002/cpu.shares 
32768
{noformat}

I have 8 of the second type of container (32768 cpu shares) on an 8 core machine. When running
8 * 32768 and 1 * 1024, I get a top that looks like this:

{noformat}
 1404 criccomi  20   0 1022m 108m  13m S 98.1  0.2   2:33.03 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Xmx160M -XX:+PrintGCDateStamps -Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0005/container_1371061837111_0005_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
 3192 criccomi  20   0 1022m 109m  13m S 98.1  0.2   2:25.93 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Xmx160M -XX:+PrintGCDateStamps -Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0009/container_1371061837111_0009_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
  428 criccomi  20   0 1022m 109m  13m S 97.7  0.2   2:36.41 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Xmx160M -XX:+PrintGCDateStamps -Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0004/container_1371061837111_0004_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
 3022 criccomi  20   0 1022m 110m  13m S 97.2  0.2   2:29.74 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Xmx160M -XX:+PrintGCDateStamps -Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0007/container_1371061837111_0007_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
32443 criccomi  20   0 1022m 109m  13m S 95.1  0.2   2:40.17 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Xmx160M -XX:+PrintGCDateStamps -Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0003/container_1371061837111_0003_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
 2850 criccomi  20   0 1022m 107m  13m S 93.6  0.2   2:31.09 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Xmx160M -XX:+PrintGCDateStamps -Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0006/container_1371061837111_0006_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
 3112 criccomi  20   0 1022m 108m  13m S 93.2  0.2   2:25.54 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Xmx160M -XX:+PrintGCDateStamps -Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0008/container_1371061837111_0008_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
31038 criccomi  20   0 1022m 109m  13m S 84.5  0.2   3:07.39 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Xmx160M -XX:+PrintGCDateStamps -Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0002/container_1371061837111_0002_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
29451 criccomi  20   0 1925m 249m  13m S 16.3  0.4   0:33.29 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Dproc_nodemanager -Xmx1000m -server -Dhadoop.log.dir=/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs
-Dyarn.log.dir=/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs -Dhadoop.log.file=yarn.log
-Dyarn.log.file=yarn.lo
30447 criccomi  20   0 1022m 109m  13m S  3.7  0.2   1:28.42 /export/apps/jdk/JDK-1_6_0_27/bin/java
-Xmx160M -XX:+PrintGCDateStamps -Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0001/container_1371061837111_0001_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
{noformat}

The column that starts with 98 is the CPU column. As you can see, container_1371061837111_0001_01_000002
is only taking 3% CPU, while the other processes are taking 100%. They're all doing the same
thing to burn up CPU, so it appears the CGroups are throttling my 1024 container as expected.

Cheers,
Chris
                
> Hook up cgroups CPU settings to the number of virtual cores allocated
> ---------------------------------------------------------------------
>
>                 Key: YARN-600
>                 URL: https://issues.apache.org/jira/browse/YARN-600
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager, scheduler
>    Affects Versions: 2.0.3-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: YARN-600.patch
>
>
> YARN-3 introduced CPU isolation and monitoring through cgroups.  YARN-2 and introduced
CPU scheduling in the capacity scheduler, and YARN-326 will introduce it in the fair scheduler.
 The number of virtual cores allocated to a container should be used to weight the number
of cgroups CPU shares given to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message