hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Macdonald (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2862) [HOD] Support PBS env vars in hod configuration
Date Sat, 02 Aug 2008 11:59:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619207#action_12619207

Craig Macdonald commented on HADOOP-2862:

With HOD from 0.17.1 this works as expected with no changes, at least for the following parameters:

temp-dir                        = /scratch/pbstmp.$PBS_JOBID
work-dirs                       = /scratch/pbstmp.$PBS_JOBID/1,/scratch/pbstmp.$PBS_JOBID/2
temp-dir                        = /scratch/pbstmp.$PBS_JOBID

Apologies, I thought I had checked this before filing the JIRA. Instead, the documentation
should be amended to note that this solution is possible. If this solution is deemed preferable,
the default configuration files should also be amended.

The added bonus of using a different partition from /tmp is that the node doesnt stop working
when / is filled ;-)

> [HOD] Support PBS env vars in hod configuration
> -----------------------------------------------
>                 Key: HADOOP-2862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2862
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/hod
>    Affects Versions: 0.16.0
>         Environment: Torque PBS
>            Reporter: Craig Macdonald
> In some batch environments, eg using Torque PBS, scratch spaces are provided on cluster
nodes for where jobs should put their temporary files. These are automatically cleaned up
when the job exists by an epilogue script.
> For instance, in our local Torque cluster, all nodes have a /scratch partition. For each
job, the prologue script creates a scratch folder owned by the user at /scratch/pbstmp.$PBS_JOBID
- $PBS_JOBID is then the env var containing the job id, as set by pbs_mom.
> Would it be possible to use these env vars in the configuration of hod. For instance,
say I want to create an hdfs on demand using hod, but that the hdfs space should be in /scratch/pbstmp.$PBS_JOBID,
not in /tmp/hod say. This would involve HOD supporting env vars in configuration, but knowing
when to substitute the env var with it's current value (ie not until running on the correct
node where the operation should take place).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message