hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-482) FS: Extend SchedulingMode to intermediate queues
Date Thu, 18 Apr 2013 00:29:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634676#comment-13634676
] 

Alejandro Abdelnur commented on YARN-482:
-----------------------------------------

Karthik, i've run the tests from the patch locally and the JVM is crashing, can you please
check what may be the problem.

I've run:

{code}
$ mvn test -Dtest=TestFairScheduler,TestComputeFairShares,TestSchedulingPolicy,TestSchedulingModemvn
-X
{code}

Got:

{code}
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Forking command line: /bin/sh -c cd /Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
&& /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Xmx1024m
-XX:+HeapDumpOnOutOfMemoryError -jar /Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefirebooter5486352351438243241.jar
/Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire4968832785999611492tmp
/Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire_01026711510169742052tmp
Running org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestComputeFairShares
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.302 sec
Forking command line: /bin/sh -c cd /Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
&& /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Xmx1024m
-XX:+HeapDumpOnOutOfMemoryError -jar /Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefirebooter6956429096288379227.jar
/Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire3733324656903328591tmp
/Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire_11280932507923501382tmp
Running org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Forking command line: /bin/sh -c cd /Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
&& /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Xmx1024m
-XX:+HeapDumpOnOutOfMemoryError -jar /Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefirebooter4319097526565824667.jar
/Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire3968109606310307351tmp
/Users/tucu/src/apache/hadoop/svn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/surefire/surefire_29016085273868453729tmp
Running org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingPolicy
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.272 sec

Results :

Tests run: 11, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................ SUCCESS [0.335s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [0.349s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [0.432s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [0.093s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.094s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [1.362s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [0.128s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [0.097s]
[INFO] Apache Hadoop Common .............................. SUCCESS [4.521s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.066s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [9.140s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [0.164s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [0.345s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.022s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.107s]
[INFO] hadoop-yarn-api ................................... SUCCESS [2.293s]
[INFO] hadoop-yarn-common ................................ SUCCESS [0.675s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.125s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [0.502s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [0.434s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [0.243s]
[INFO] hadoop-yarn-server-resourcemanager ................ FAILURE [4.374s]
[INFO] hadoop-yarn-server-tests .......................... SKIPPED
[INFO] hadoop-yarn-client ................................ SKIPPED
[INFO] hadoop-yarn-applications .......................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED
[INFO] hadoop-yarn-site .................................. SKIPPED
[INFO] hadoop-yarn-project ............................... SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
[INFO] Apache Hadoop Distributed Copy .................... SKIPPED
[INFO] Apache Hadoop Archives ............................ SKIPPED
[INFO] Apache Hadoop Rumen ............................... SKIPPED
[INFO] Apache Hadoop Gridmix ............................. SKIPPED
[INFO] Apache Hadoop Data Join ........................... SKIPPED
[INFO] Apache Hadoop Extras .............................. SKIPPED
[INFO] Apache Hadoop Pipes ............................... SKIPPED
[INFO] Apache Hadoop Tools Dist .......................... SKIPPED
[INFO] Apache Hadoop Tools ............................... SKIPPED
[INFO] Apache Hadoop Distribution ........................ SKIPPED
[INFO] Apache Hadoop Client .............................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 27.073s
[INFO] Finished at: Wed Apr 17 17:28:32 PDT 2013
[INFO] Final Memory: 40M/150M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-server-resourcemanager: ExecutionException; nested exception
is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated
without saying properly goodbye. VM crash or System.exit called ? -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test) on project hadoop-yarn-server-resourcemanager: ExecutionException; nested exception
is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated
without saying properly goodbye. VM crash or System.exit called ?
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.MojoExecutionException: ExecutionException; nested exception
is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated
without saying properly goodbye. VM crash or System.exit called ?
	at org.apache.maven.plugin.surefire.SurefirePlugin.assertNoException(SurefirePlugin.java:160)
	at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:149)
	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:650)
	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:586)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
	... 19 more
Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException;
nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The
forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:192)
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:121)
	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:740)
	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAllProviders(AbstractSurefireMojo.java:682)
	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:648)
	... 22 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked
VM terminated without saying properly goodbye. VM crash or System.exit called ?
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:176)
	... 26 more
Caused by: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye.
VM crash or System.exit called ?
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.close(ForkClient.java:257)
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:301)
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$100(ForkStarter.java:75)
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:163)
	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:160)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:680)
[ERROR] 
{code}

                
> FS: Extend SchedulingMode to intermediate queues
> ------------------------------------------------
>
>                 Key: YARN-482
>                 URL: https://issues.apache.org/jira/browse/YARN-482
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>    Affects Versions: 2.0.3-alpha
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>         Attachments: yarn-482.patch, yarn-482.patch, yarn-482.patch, yarn-482.patch,
yarn-482.patch
>
>
> FS allows setting {{SchedulingMode}} for leaf queues. Extending this to non-leaf queues
allows using different kinds of fairness: e.g., root can have three child queues - fair-mem,
drf-cpu-mem, drf-cpu-disk-mem taking different number of resources into account. In turn,
this allows users to decide on the scheduling latency vs sophistication of the scheduling
mode.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message