hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Bacsko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-9482) DistributedShell job with localization fails in unsecure cluster
Date Wed, 17 Apr 2019 10:18:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819930#comment-16819930
] 

Peter Bacsko commented on YARN-9482:
------------------------------------

[~Prabhu Joseph] wouldn't it be better wrap filesystem calls in {{doAs}} calls? That sould
always work - in a secure cluster, containers started on behalf of the submitter, not as "yarn"
(unless overridden with {{yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users}}).
In a nonsecure cluster, you can use {{doAs}} without limitations.

But anyway since this is a test application, I'm OK with using {{/tmp}} as a staging dir.

Minor thing:  {{new Path(STAGING_AREA + Path.SEPARATOR + appId + Path.SEPARATOR)}} could be
simplified to {{new Path(STAGING_AREA, appId)}}

> DistributedShell job with localization fails in unsecure cluster
> ----------------------------------------------------------------
>
>                 Key: YARN-9482
>                 URL: https://issues.apache.org/jira/browse/YARN-9482
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: distributed-shell
>    Affects Versions: 3.3.0
>            Reporter: Prabhu Joseph
>            Assignee: Prabhu Joseph
>            Priority: Major
>         Attachments: YARN-9482-001.patch, YARN-9482-002.patch
>
>
> DistributedShell job with localization fails in unsecure cluster. The client localizes
the input files to home directory (job user) whereas the AM runs as yarn user reads from it's
home directory.
> *Command:*
> {code}
> yarn jar /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
-shell_command ls  -shell_args / -jar  /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
-localize_files /tmp/prabhu
> {code}
> {code}
> Exception in thread "Thread-4" java.io.UncheckedIOException: Error during localization
setup
> 	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1495)
> 	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> 	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
> 	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.run(ApplicationMaster.java:1481)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://yarn-ats-1:8020/user/yarn/DistributedShell/application_1554817981283_0003/prabhu
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1586)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
> 	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
> 	at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1487)
> {code}	



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message