hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "haosdent@gmail.com" <haosd...@gmail.com>
Subject Re: Running TestLinuxContainerExecutor Failed
Date Mon, 28 Oct 2013 15:33:22 GMT
oh, I checkout the code from trunk. How I register "file://"? Thank you very much.




Bobby Evans <evans@yahoo-inc.com>编写:

>Wow, I am not totally sure what is going on here but somehow file:// is
>not registered with Hadoop in your config.  I have not run the tests in a
>while.  What version of hadoop are you using?
>
>--Bobby
>
>On 10/28/13 7:37 AM, "Haosong Huang" <haosdent@gmail.com> wrote:
>
>>Hi, I have done the prepare work described in comments of
>>TestLinuxContainerExecutor, but I run it failed. If the comments
>>omitted any improtant steps?
>>
>>My System: Ubuntu 13.04 Linux Kernel:3.8.0-26
>>
>>The comments:
>>/**
>> * This is intended to test the LinuxContainerExecutor code, but because
>>of
>> * some security restrictions this can only be done with some special
>>setup
>> * first.
>> * <br><ol>
>> * <li>Compile the code with container-executor.conf.dir set to the
>>location you
>> * want for testing.
>> * <br><pre><code>
>> * > mvn clean install -Pnative -Dcontainer-executor.conf.dir=/etc/hadoop
>> *                          -DskipTests
>> * </code></pre>
>> *
>> * <li>Set up 
>><code>${container-executor.conf.dir}/container-executor.cfg</code>
>> * container-executor.cfg needs to be owned by root and have in it the
>>proper
>> * config values.
>> * <br><pre><code>
>> * > cat /etc/hadoop/container-executor.cfg
>> * yarn.nodemanager.linux-container-executor.group=mapred
>> * #depending on the user id of the application.submitter option
>> * min.user.id=1
>> * > sudo chown root:root /etc/hadoop/container-executor.cfg
>> * > sudo chmod 444 /etc/hadoop/container-executor.cfg
>> * </code></pre>
>> *
>> * <li>Move the binary and set proper permissions on it. It needs to be
>>owned
>> * by root, the group needs to be the group configured in
>>container-executor.cfg,
>> * and it needs the setuid bit set. (The build will also overwrite it so
>>you
>> * need to move it to a place that you can support it.
>> * <br><pre><code>
>> * > cp 
>>./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-no
>>demanager/src/main/c/container-executor/container-executor
>>/tmp/
>> * > sudo chown root:mapred /tmp/container-executor
>> * > sudo chmod 4550 /tmp/container-executor
>> * </code></pre>
>> *
>> * <li>Run the tests with the execution enabled (The user you run the
>>tests as
>> * needs to be part of the group from the config.
>> * <br><pre><code>
>> * mvn test -Dtest=TestLinuxContainerExecutor
>>-Dapplication.submitter=nobody
>>-Dcontainer-executor.path=/tmp/container-executor
>> * </code></pre>
>> * </ol>
>> */
>>
>>The error logs:
>>
>>2013-10-28 18:29:40,326 INFO  [main]
>>nodemanager.TestLinuxContainerExecutor
>>(TestLinuxContainerExecutor.java:setup(122)) - Setting
>>yarn.nodemanager.linux-container-executor.path=/tmp/container-executor
>>NM_DIR:/home/haosdent/workspace/java/hadoop-apache/target/org.apache.hadoo
>>p.yarn.server.nodemanager.TestLinuxContainerExecutor-workSpace/logDir
>>2013-10-28 18:29:40,341 ERROR [main] security.UserGroupInformation
>>(UserGroupInformation.java:doAs(1511)) - PriviledgedActionException
>>as:haosdent (auth:SIMPLE)
>>cause:org.apache.hadoop.fs.UnsupportedFileSystemException: No
>>AbstractFileSystem for scheme: file
>>java.security.PrivilegedActionException:
>>org.apache.hadoop.fs.UnsupportedFileSystemException: No
>>AbstractFileSystem for scheme: file
>>at java.security.AccessController.doPrivileged(Native Method)
>>at javax.security.auth.Subject.doAs(Subject.java:396)
>>at 
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>>java:1508)
>>at 
>>org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:31
>>9)
>>at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:433)
>>at 
>>org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:47
>>0)
>>at 
>>org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
>>nit(LocalDirsHandlerService.java:133)
>>at 
>>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>at 
>>org.apache.hadoop.yarn.server.nodemanager.TestLinuxContainerExecutor.setup
>>(TestLinuxContainerExecutor.java:132)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at 
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>39)
>>at 
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>pl.java:25)
>>at java.lang.reflect.Method.invoke(Method.java:597)
>>at 
>>org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMetho
>>d.java:45)
>>at 
>>org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable
>>.java:15)
>>at 
>>org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.
>>java:42)
>>at 
>>org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:
>>27)
>>at 
>>org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30
>>)
>>at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>>at 
>>org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
>>ava:68)
>>at 
>>org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
>>ava:47)
>>at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>>at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>>at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>>at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>>at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>>at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>>at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
>>at 
>>com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTes
>>tRunner.java:77)
>>at 
>>com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitS
>>tarter.java:195)
>>at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at 
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>39)
>>at 
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>pl.java:25)
>>at java.lang.reflect.Method.invoke(Method.java:597)
>>at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
>>Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
>>AbstractFileSystem for scheme: file
>>at 
>>org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSyste
>>m.java:152)
>>at 
>>org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:240)
>>at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:322)
>>at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:319)
>>... 36 more
>>2013-10-28 18:29:40,348 INFO  [main] service.AbstractService
>>(AbstractService.java:noteFailure(272)) - Service
>>org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService
>>failed in state INITED; cause:
>>org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to get
>>the local filesystem
>>org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to get
>>the local filesystem
>>at 
>>org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
>>nit(LocalDirsHandlerService.java:135)
>>at 
>>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>at 
>>org.apache.hadoop.yarn.server.nodemanager.TestLinuxContainerExecutor.setup
>>(TestLinuxContainerExecutor.java:132)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at 
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>39)
>>at 
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>pl.java:25)
>>at java.lang.reflect.Method.invoke(Method.java:597)
>>at 
>>org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMetho
>>d.java:45)
>>at 
>>org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable
>>.java:15)
>>at 
>>org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.
>>java:42)
>>at 
>>org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:
>>27)
>>at 
>>org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30
>>)
>>at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>>at 
>>org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
>>ava:68)
>>at 
>>org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
>>ava:47)
>>at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>>at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>>at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>>at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>>at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>>at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>>at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
>>at 
>>com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTes
>>tRunner.java:77)
>>at 
>>com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitS
>>tarter.java:195)
>>at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at 
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>39)
>>at 
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>pl.java:25)
>>at java.lang.reflect.Method.invoke(Method.java:597)
>>at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
>>Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
>>AbstractFileSystem for scheme: file
>>at 
>>org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSyste
>>m.java:152)
>>at 
>>org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:240)
>>at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:322)
>>at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:319)
>>at java.security.AccessController.doPrivileged(Native Method)
>>at javax.security.auth.Subject.doAs(Subject.java:396)
>>at 
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>>java:1508)
>>at 
>>org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:31
>>9)
>>at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:433)
>>at 
>>org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:47
>>0)
>>at 
>>org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
>>nit(LocalDirsHandlerService.java:133)
>>... 29 more
>>
>>org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to get
>>the local filesystem
>>at 
>>org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
>>nit(LocalDirsHandlerService.java:135)
>>at 
>>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>at 
>>org.apache.hadoop.yarn.server.nodemanager.TestLinuxContainerExecutor.setup
>>(TestLinuxContainerExecutor.java:132)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at 
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>39)
>>at 
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>pl.java:25)
>>at 
>>org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMetho
>>d.java:45)
>>at 
>>org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable
>>.java:15)
>>at 
>>org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.
>>java:42)
>>at 
>>org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:
>>27)
>>at 
>>org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30
>>)
>>at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>>at 
>>org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
>>ava:68)
>>at 
>>org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.j
>>ava:47)
>>at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>>at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>>at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>>at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>>at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>>at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>>at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
>>at 
>>com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTes
>>tRunner.java:77)
>>at 
>>com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitS
>>tarter.java:195)
>>at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at 
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>39)
>>at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
>>Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
>>AbstractFileSystem for scheme: file
>>at 
>>org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSyste
>>m.java:152)
>>at 
>>org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:240)
>>at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:322)
>>at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:319)
>>at java.security.AccessController.doPrivileged(Native Method)
>>at javax.security.auth.Subject.doAs(Subject.java:396)
>>at 
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>>java:1508)
>>at 
>>org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:31
>>9)
>>at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:433)
>>at 
>>org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:47
>>0)
>>at 
>>org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceI
>>nit(LocalDirsHandlerService.java:133)
>>... 29 more
>>
>>
>>Process finished with exit code 255
>>
>>
>>-- 
>>Best Regards,
>>Haosdent Huang
>
Mime
View raw message