From yarn-issues-return-166743-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Wed Apr 17 07:45:07 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id E66B0180672 for ; Wed, 17 Apr 2019 09:45:06 +0200 (CEST) Received: (qmail 67511 invoked by uid 500); 17 Apr 2019 07:45:06 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 67500 invoked by uid 99); 17 Apr 2019 07:45:06 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Apr 2019 07:45:06 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 964D4E27D4 for ; Wed, 17 Apr 2019 07:45:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 4952E2459C for ; Wed, 17 Apr 2019 07:45:00 +0000 (UTC) Date: Wed, 17 Apr 2019 07:45:00 +0000 (UTC) From: "Prabhu Joseph (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-9482) DistributedShell job with localization fails in unsecure cluster MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819808#comment-16819808 ] Prabhu Joseph commented on YARN-9482: ------------------------------------- [~pbacsko] Can you review this jira as well when you get time. This fixes localization failure in unsecure cluster. > DistributedShell job with localization fails in unsecure cluster > ---------------------------------------------------------------- > > Key: YARN-9482 > URL: https://issues.apache.org/jira/browse/YARN-9482 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-shell > Affects Versions: 3.3.0 > Reporter: Prabhu Joseph > Assignee: Prabhu Joseph > Priority: Major > Attachments: YARN-9482-001.patch, YARN-9482-002.patch > > > DistributedShell job with localization fails in unsecure cluster. The client localizes the input files to home directory (job user) whereas the AM runs as yarn user reads from it's home directory. > *Command:* > {code} > yarn jar /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar -shell_command ls -shell_args / -jar /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar -localize_files /tmp/prabhu > {code} > {code} > Exception in thread "Thread-4" java.io.UncheckedIOException: Error during localization setup > at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1495) > at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) > at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.run(ApplicationMaster.java:1481) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.FileNotFoundException: File does not exist: hdfs://yarn-ats-1:8020/user/yarn/DistributedShell/application_1554817981283_0003/prabhu > at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1586) > at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579) > at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594) > at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1487) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org