hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haosong Huang <haosd...@gmail.com>
Subject Re: Running TestLinuxContainerExecutor Failed
Date Mon, 28 Oct 2013 17:18:01 GMT
And I have another question. Because of the addition configure before run
unit tests, Jenkins didn't test TestLinuxContainerExecutor and pass it in
fact. Is it right?


2013/10/28 haosdent@gmail.com <haosdent@gmail.com>

> oh, I checkout the code from trunk. How I register "file://"? Thank you
> very much.
>
>
>
>
>
> Bobby Evans <evans@yahoo-inc.com>编写:
>
>
> Wow, I am not totally sure what is going on here but somehow file:// is
> not registered with Hadoop in your config.  I have not run the tests in a
> while.  What version of hadoop are you using?
>
> --Bobby
>
> On 10/28/13 7:37 AM, "Haosong Huang" <haosdent@gmail.com> wrote:
>
> >Hi, I have done the prepare work described in comments of
> >TestLinuxContainerExecutor, but I run it failed. If the comments
> >omitted any improtant steps?
> >
> >My System: Ubuntu 13.04 Linux Kernel:3.8.0-26
> >
> >The comments:
> >/**
> > * This is intended to test the LinuxContainerExecutor code, but because
> >of
> > * some security restrictions this can only be done with some special
> >setup
> > * first.
> > * <br><ol>
> > * <li>Compile the code with container-executor.conf.dir set to the
> >location you
> > * want for testing.
> > * <br><pre><code>
> > * > mvn clean install -Pnative -Dcontainer-executor.conf.dir=/etc/hadoop
> > *                          -DskipTests
> > * </code></pre>
> > *
> > * <li>Set up
> ><code>${container-executor.conf.dir}/container-executor.cfg</code>
> > * container-executor.cfg needs to be owned by root and have in it the
> >proper
> > * config values.
> > * <br><pre><code>
> > * > cat /etc/hadoop/container-executor.cfg
> > * yarn.nodemanager.linux-container-executor.group=mapred
> > * #depending on the user id of the application.submitter option
> > * min.user.id=1
> > * > sudo chown root:root /etc/hadoop/container-executor.cfg
> > * > sudo chmod 444 /etc/hadoop/container-executor.cfg
> > * </code></pre>
> > *
> > * <li>Move the binary and set proper permissions on it. It needs to be
> >owned
> > * by root, the group needs to be the group configured in
> >container-executor.cfg,
> > * and it needs the setuid bit set. (The build will also overwrite it so
> >you
> > * need to move it to a place that you can support it.
> > * <br><pre><code>
> > * > cp
> >./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-no
> >demanager/src/main/c/container-executor/container-executor
> >/tmp/
> > * > sudo chown root:mapred /tmp/container-executor
> > * > sudo chmod 4550 /tmp/container-executor
> > * </code></pre>
> > *
> > * <li>Run the tests with the execution enabled (The user you run the
> >tests as
> > * needs to be part of the group from the config.
> > * <br><pre><code>
> > * mvn test -Dtest=TestLinuxContainerExecutor
> >-Dapplication.submitter=nobody
> >-Dcontainer-executor.path=/tmp/container-executor
> > * </code></pre>
> > * </ol>
> > */
> >
> >The error logs:
> >
> >2013-10-28 18:29:40,326 INFO  [main]
> >nodemanager.TestLinuxContainerExecutor
> >(TestLinuxContainerExecutor.java:setup(122)) - Setting
> >yarn.nodemanager.linux-container-executor.path=/tmp/container-executor
> >NM_DIR:/home/haosdent/workspace/java/hadoop-apache/target/org.apache.hadoo
> >p.yarn.server.nodemanager.TestLinuxContainerExecutor-workSpace/logDir
> >2013-10-28 18:29:40,341 ERROR [main] security.UserGroupInformation
> >(UserGroupInformation.java:doAs(1511)) - PriviledgedActionException
> >as:haosdent (auth:SIMPLE)
> >cause:org.apache.hadoop.fs.UnsupportedFileSystemException: No
> >AbstractFileSystem for scheme: file
> >java.security.PrivilegedActionException:
> >org.apache.hadoop.fs.UnsupportedFileSystemException: No
> >AbstractFileSystem for scheme: file
> >at java.security.AccessController.doPrivileged(Native Method)
> >at javax.security.auth.Subject.doAs(Subject.java:396)
> >at
> >org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
> >java:1508)
> >at
> >org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:31
> >9)
> >at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:433)
> >at
> >org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:47
> >0)
> >at
> >org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
> >nit(LocalDirsHandlerService.java:133)
> >at
> >org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> >at
> >org.apache.hadoop.yarn.server.nodemanager.TestLinuxContainerExecutor.setup
> >(TestLinuxContainerExecutor.java:132)
> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >39)
> >at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >pl.java:25)
> >at java.lang.reflect.Method.invoke(Method.java:597)
> >at
> >org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMetho
> >d.java:45)
> >at
> >org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable
> >.java:15)
> >at
> >org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.
> >java:42)
> >at
> >org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:
> >27)
> >at
> >org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30
> >)
> >at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> >at
> >org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
> >ava:68)
> >at
> >org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
> >ava:47)
> >at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> >at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> >at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> >at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> >at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> >at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> >at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> >at
> >com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTes
> >tRunner.java:77)
> >at
> >com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitS
> >tarter.java:195)
> >at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >39)
> >at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >pl.java:25)
> >at java.lang.reflect.Method.invoke(Method.java:597)
> >at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
> >Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
> >AbstractFileSystem for scheme: file
> >at
> >org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSyste
> >m.java:152)
> >at
> >org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:240)
> >at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:322)
> >at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:319)
> >... 36 more
> >2013-10-28 18:29:40,348 INFO  [main] service.AbstractService
> >(AbstractService.java:noteFailure(272)) - Service
> >org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService
> >failed in state INITED; cause:
> >org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to get
> >the local filesystem
> >org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to get
> >the local filesystem
> >at
> >org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
> >nit(LocalDirsHandlerService.java:135)
> >at
> >org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> >at
> >org.apache.hadoop.yarn.server.nodemanager.TestLinuxContainerExecutor.setup
> >(TestLinuxContainerExecutor.java:132)
> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >39)
> >at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >pl.java:25)
> >at java.lang.reflect.Method.invoke(Method.java:597)
> >at
> >org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMetho
> >d.java:45)
> >at
> >org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable
> >.java:15)
> >at
> >org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.
> >java:42)
> >at
> >org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:
> >27)
> >at
> >org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30
> >)
> >at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> >at
> >org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
> >ava:68)
> >at
> >org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
> >ava:47)
> >at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> >at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> >at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> >at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> >at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> >at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> >at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> >at
> >com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTes
> >tRunner.java:77)
> >at
> >com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitS
> >tarter.java:195)
> >at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >39)
> >at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >pl.java:25)
> >at java.lang.reflect.Method.invoke(Method.java:597)
> >at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
> >Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
> >AbstractFileSystem for scheme: file
> >at
> >org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSyste
> >m.java:152)
> >at
> >org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:240)
> >at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:322)
> >at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:319)
> >at java.security.AccessController.doPrivileged(Native Method)
> >at javax.security.auth.Subject.doAs(Subject.java:396)
> >at
> >org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
> >java:1508)
> >at
> >org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:31
> >9)
> >at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:433)
> >at
> >org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:47
> >0)
> >at
> >org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
> >nit(LocalDirsHandlerService.java:133)
> >... 29 more
> >
> >org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to get
> >the local filesystem
> >at
> >org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
> >nit(LocalDirsHandlerService.java:135)
> >at
> >org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> >at
> >org.apache.hadoop.yarn.server.nodemanager.TestLinuxContainerExecutor.setup
> >(TestLinuxContainerExecutor.java:132)
> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >39)
> >at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >pl.java:25)
> >at
> >org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMetho
> >d.java:45)
> >at
> >org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable
> >.java:15)
> >at
> >org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.
> >java:42)
> >at
> >org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:
> >27)
> >at
> >org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30
> >)
> >at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> >at
> >org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
> >ava:68)
> >at
> >org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
> >ava:47)
> >at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> >at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> >at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> >at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> >at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> >at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> >at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> >at
> >com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTes
> >tRunner.java:77)
> >at
> >com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitS
> >tarter.java:195)
> >at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >39)
> >at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
> >Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
> >AbstractFileSystem for scheme: file
> >at
> >org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSyste
> >m.java:152)
> >at
> >org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:240)
> >at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:322)
> >at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:319)
> >at java.security.AccessController.doPrivileged(Native Method)
> >at javax.security.auth.Subject.doAs(Subject.java:396)
> >at
> >org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
> >java:1508)
> >at
> >org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:31
> >9)
> >at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:433)
> >at
> >org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:47
> >0)
> >at
> >org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
> >nit(LocalDirsHandlerService.java:133)
> >... 29 more
> >
> >
> >Process finished with exit code 255
> >
> >
> >--
> >Best Regards,
> >Haosdent Huang
>
>
>


-- 
Best Regards,
Haosdent Huang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message