hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "WritingYarnApps" by ChrisRiccomini
Date Wed, 21 Sep 2011 04:04:59 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "WritingYarnApps" page has been changed by ChrisRiccomini:

  '''How can I distribute my application's jars distributed to all of the nodes in the YARN
cluster that need it?'''
+ You can use the LocalResource to add resources to your application request. This will cause
YARN to distribute the resource to the application master node, and all of the the container
nodes. If the resource is a tgz, zip, or jar, you can have YARN unzip it. Then, all you need
to do is add the unzipped folder to your classpath. For example, when creating your application
+ {{{    val packageFile = new File(packagePath);
+     val packageUrl = ConverterUtils.getYarnUrlFromPath(FileContext.getFileContext.makeQualified(new
+     packageResource.setResource(packageUrl);
+     packageResource.setSize(packageFile.length());
+     packageResource.setTimestamp(packageFile.lastModified());
+     packageResource.setType(LocalResourceType.ARCHIVE);
+     packageResource.setVisibility(LocalResourceVisibility.APPLICATION);
+     resource.setMemory(memory)
+     containerCtx.setResource(resource)
+     containerCtx.setCommands(ImmutableList.of(new ApplicationMasterExecutor(packagePath)
+       .addCommand("java -cp './package/*' some.class.to.Run "
+         + "1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout "
+         + "2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr")
+       .execute(new Configuration))
+     containerCtx.setLocalResources(Collections.singletonMap("package", packageResource))
+     appCtx.setApplicationId(appId)
+     appCtx.setUser(user.getShortUserName)
+     appCtx.setAMContainerSpec(containerCtx)
+     request.setApplicationSubmissionContext(appCtx)
+     applicationsManager.submitApplication(request)}}}
+ As you can see, the setLocalResources command takes a map of names to resources. The name
becomes a sym link in your application's working directory, so you can just refer to the artifacts
inside by using ./package/*. ''Note: Java's classpath (cp) argument is VERY sensitive. Make
sure you get the syntax EXACTLY correct.''
  '''Why does my application's client have to send the app ID/attempt ID/cluster timestamp
to my application master?'''
  This is actually going to be improved. It will likely become environment variables, or substituted
variables (like the fail count). For more details, track this ticket: https://issues.apache.org/jira/browse/MAPREDUCE-3055

View raw message