hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "WritingYarnApps" by ChrisRiccomini
Date Wed, 21 Sep 2011 17:40:51 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "WritingYarnApps" page has been changed by ChrisRiccomini:
http://wiki.apache.org/hadoop/WritingYarnApps?action=diff&rev1=10&rev2=11

  
  '''How can I distribute my application's jars to all of the nodes in the YARN cluster that
need it?'''
  
- You can use the LocalResource to add resources to your application request. This will cause
YARN to distribute the resource to the application master node, and all of the the container
nodes. If the resource is a tgz, zip, or jar, you can have YARN unzip it. Then, all you need
to do is add the unzipped folder to your classpath. For example, when creating your application
request:
+ You can use the LocalResource to add resources to your application request. This will cause
YARN to distribute the resource to the application master node. If the resource is a tgz,
zip, or jar, you can have YARN unzip it. Then, all you need to do is add the unzipped folder
to your classpath. For example, when creating your application request:
  
  {{{
      File packageFile = new File(packagePath);
@@ -48, +48 @@

  
  As you can see, the setLocalResources command takes a map of names to resources. The name
becomes a sym link in your application's working directory, so you can just refer to the artifacts
inside by using ./package/*. ''Note: Java's classpath (cp) argument is VERY sensitive. Make
sure you get the syntax EXACTLY correct.''
  
+ Once your package is distributed to your application master, you'll need to follow the same
process whenever your application master starts a new container (assuming you want the resources
to be sent to your container). The code for this is the same. You just need to make sure that
you give your application master the package path (either HDFS, or local), so that it can
send the resource URL along with the container ctx.
+ 
  '''Why does my application's client have to send the app ID/attempt ID/cluster timestamp
to my application master?'''
  
  This is actually going to be improved. It will likely become environment variables, or substituted
variables (like the fail count). For more details, track this ticket: https://issues.apache.org/jira/browse/MAPREDUCE-3055

Mime
View raw message