hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: setLocalResources() on ContainerLaunchContext
Date Mon, 05 Aug 2013 09:30:52 GMT
The string for each LocalResource in the map can be anything that
serves as a common identifier name for your application. At execution
time, the passed resource filename will be aliased to the name you've
mapped it to, so that the application code need not track special
names. The behavior is very similar to how you can, in MR, define a
symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar).

For an example, checkout the DistributedShell app sources.

Over [1], you can see we take a user provided file path to a shell
script. This can be named anything as it is user-supplied.
Onto [2], we define this as a local resource [2.1] and embed it with a
different name (the string you ask about) [2.2], as defined at [3] as
an application reference-able constant.
Note that in [4], we add to the Container arguments the aliased name
we mapped it to (i.e. [3]) and not the original filename we received
from the user. The resource is placed on the container with this name
instead, so thats what we choose to execute.

[1] - https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390

[2] - [2.1] https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764
and [2.2] https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780

[3] - https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205

[4] - https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791

On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri
<write2kishore@gmail.com> wrote:
> Hi,
>   Can someone please tell me what is the use of calling setLocalResources()
> on ContainerLaunchContext?
>   And, also an example of how to use this will help...
>  I couldn't guess what is the String in the map that is passed to
> setLocalResources() like below:
>       // Set the local resources
>       Map<String, LocalResource> localResources = new HashMap<String,
> LocalResource>();
> Thanks,
> Kishore

Harsh J

View raw message