Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B3F1210993 for ; Mon, 2 Dec 2013 23:28:22 +0000 (UTC) Received: (qmail 80223 invoked by uid 500); 2 Dec 2013 23:28:17 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 80042 invoked by uid 500); 2 Dec 2013 23:28:17 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 80035 invoked by uid 99); 2 Dec 2013 23:28:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Dec 2013 23:28:17 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of acm@hortonworks.com designates 209.85.160.45 as permitted sender) Received: from [209.85.160.45] (HELO mail-pb0-f45.google.com) (209.85.160.45) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Dec 2013 23:28:12 +0000 Received: by mail-pb0-f45.google.com with SMTP id rp16so20047174pbb.18 for ; Mon, 02 Dec 2013 15:27:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:message-id:mime-version:subject:date :references:to:in-reply-to:content-type; bh=ZCTyiuIxxHEvcpvxDdQlLUJ5BQqO5VUwmn0UFamjqdU=; b=HFqszXCHezn9bgTGl9vaDJ4chtCbbm/+KzFnFFlSlXSEAGsTv8zynKvg1OI4S+YZsH EqQwNUURGJLfoQPMwAcD9hOEoxNmR+B/Oou/gUoAw0BknpNJbYKucjie983x9Z2LvOV/ 4hf/4RIMjvrUBb0Ve/crZmVNSdkfq3fvIT4UpTS040JBtu0EblLACVzV9KDrssjXDKi9 xppaWA7usSeVKl3Nc5pjJGOe99a3t1GDFJOCeq9OPlb+sJ+20pxAKpyKnpSUDhwm7qDy jfHsCVsIvjTXwR5rDpm85FlcP1Qy115VgA0SlCTfMMY68i5WafYJyxDCya1877cbfWnF yBJA== X-Gm-Message-State: ALoCoQlD9O8IQVkNIc0WFOpKG5YtFMIJ7zX+bFhlySd4A3SIL0EvqniAk7kMsJvJdX4QFUODS0hF9Mhk0Vr2FbCCvSwpSzhNNpk6FdYAxQniTDFACJ/ckN4= X-Received: by 10.69.12.36 with SMTP id en4mr34480310pbd.54.1386026872439; Mon, 02 Dec 2013 15:27:52 -0800 (PST) Received: from [10.11.3.119] ([192.175.27.2]) by mx.google.com with ESMTPSA id bp5sm125246157pbb.18.2013.12.02.15.27.50 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 02 Dec 2013 15:27:51 -0800 (PST) From: Arun C Murthy Message-Id: <541BEC8B-0E40-4923-AF68-982990AC01B3@hortonworks.com> Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: YARN: LocalResources and file distribution Date: Mon, 2 Dec 2013 15:27:49 -0800 References: To: user@hadoop.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1510) Content-Type: multipart/alternative; boundary="Apple-Mail=_3B6B72E9-9795-4757-AC98-33131549D28D" X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_3B6B72E9-9795-4757-AC98-33131549D28D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1 Robert, YARN, by default, will only download *resource* from a shared namespace (e= .g. HDFS). If /home/hadoop/robert/large_jar.jar is available on each node then you ca= n specify path as file:///home/hadoop/robert/large_jar.jar and it should wo= rk. Else, you'll need to copy /home/hadoop/robert/large_jar.jar to HDFS and th= en specify hdfs://host:port/path/to/large_jar.jar. hth, Arun On Dec 1, 2013, at 12:03 PM, Robert Metzger wrote: > Hello, >=20 > I'm currently writing code to run my application using Yarn (Hadoop 2.2.0= ). > I used this code as a skeleton: https://github.com/hortonworks/simple-yar= n-app >=20 > Everything works fine on my local machine or on a cluster with the shared= directories, but when I want to access resources outside of commonly acces= sible locations, my application fails. >=20 > I have my application in a large jar file, containing everything (Submiss= ion Client, Application Master, and Workers).=20 > The submission client registers the large jar file as a local resource fo= r the Application master's context. >=20 > In my understanding, Yarn takes care of transferring the client-local res= ources to the application master's container. > This is also stated here: http://hadoop.apache.org/docs/current/hadoop-ya= rn/hadoop-yarn-site/WritingYarnApplications.html >=20 > You can use the LocalResource to add resources to your application reques= t. This will cause YARN to distribute the resource to the ApplicationMaster= node. >=20 > If I'm starting my jar from the dir "/home/hadoop/robert/large_jar.jar", = I'll get the following error from the nodemanager (another node in the clus= ter): >=20 > 2013-12-01 20:13:00,810 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.localizer.ResourceLocalizationService: Failed to download rs= rc { { file:/home/hadoop/robert/large_jar.jar, .. >=20 > So it seems as this node tries to access the file from its local file sys= tem. >=20 > Do I have to use another "protocol" for the file, something like "file://= host:port/home/blabla" ? >=20 > Is it true that Yarn is able to distribute files (not using hdfs obviousl= y?) ? >=20 >=20 > The distributedshell-example suggests that I have to use HDFS: https://gi= thub.com/apache/hadoop-common/blob/50f0de14e377091c308c3a74ed089a7e4a7f0bfe= /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-appli= cations-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/= distributedshell/Client.java >=20 >=20 > Sincerely, > Robert >=20 >=20 >=20 >=20 >=20 -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ --=20 CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to= =20 which it is addressed and may contain information that is confidential,=20 privileged and exempt from disclosure under applicable law. If the reader= =20 of this message is not the intended recipient, you are hereby notified that= =20 any printing, copying, dissemination, distribution, disclosure or=20 forwarding of this communication is strictly prohibited. If you have=20 received this communication in error, please contact the sender immediately= =20 and delete it from your system. Thank You. --Apple-Mail=_3B6B72E9-9795-4757-AC98-33131549D28D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=ISO-8859-1 Robert,

<= div> YARN, by default, will only download *resource* from a shared nam= espace (e.g. HDFS).

 If /home/hadoop/robert/l= arge_jar.jar is available on each node then you can specify path as file://= /home/hadoop/robert/large_jar.jar and it should work.

<= div> Else, you'll need to copy /home/hadoop/robert/large_jar.jar = to HDFS and then specify hdfs://host:port/path/to/large_jar.jar.

hth,<= /div>
Arun

On Dec 1, 2013, at 12:03 PM, Robert = Metzger <metrobert@gmail.com&= gt; wrote:

Hello,

I'm currently writing code to run my = application using Yarn (Hadoop 2.2.0).
I used this code as a skeleton:&n= bsp;https://github.com/hortonworks/simple-yarn-app

Everything works fine on my local machine or on a cluster with the shar= ed directories, but when I want to access resources outside of commonly acc= essible locations, my application fails.

I have my application in a = large jar file, containing everything (Submission Client, Application Maste= r, and Workers). 
The submission client registers the large jar file as a local resource for = the Application master's context.

In my understanding, Yarn takes ca= re of transferring the client-local resources to the application master's c= ontainer.
This is also stated here: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/W= ritingYarnApplications.html

You can use the LocalResourc= e to add resources to your application request. This will cause YARN to dis= tribute the resource to the ApplicationMaster node.

If I'm starting my jar from the dir "/home/hadoop/robert/large_jar.jar", I'll get the following error= from the nodemanager (another node in the cluster):

2013-12-01 20:13:00,810 INFO org.apache.had= oop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalization= Service: Failed to download rsrc { { file:/home/hadoop/robert/large_jar.jar, ..

So= it seems as this node tries to access the file from its local file system.=

Do I have to use another "protocol" for the file, somet= hing like "file://host:port/home/= blabla" ?



Sincerely,
Robert






--
Arun C. Murthy
Ho= rtonworks Inc.
http://hortonworks.co= m/


CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --Apple-Mail=_3B6B72E9-9795-4757-AC98-33131549D28D--