Return-Path: 
 <hadoop-dev-return-3382-apmail-lucene-hadoop-dev-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org
Received: (qmail 95583 invoked from network); 28 Aug 2006 22:02:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199)
  by minotaur.apache.org with SMTP; 28 Aug 2006 22:02:52 -0000
Received: (qmail 42611 invoked by uid 500); 28 Aug 2006 22:02:45 -0000
Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org
Received: (qmail 42596 invoked by uid 500); 28 Aug 2006 22:02:45 -0000
Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hadoop-dev-help@lucene.apache.org>
List-Unsubscribe: <mailto:hadoop-dev-unsubscribe@lucene.apache.org>
List-Post: <mailto:hadoop-dev@lucene.apache.org>
List-Id: <hadoop-dev.lucene.apache.org>
Reply-To: hadoop-dev@lucene.apache.org
Delivered-To: mailing list hadoop-dev@lucene.apache.org
Received: (qmail 42579 invoked by uid 99); 28 Aug 2006 22:02:45 -0000
Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Aug 2006 15:02:45 -0700
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [209.237.227.198] (HELO brutus.apache.org) (209.237.227.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Aug 2006 15:02:44 -0700
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4F59541000E
	for <hadoop-dev@lucene.apache.org>; Mon, 28 Aug 2006 21:59:23 +0000 (GMT)
Message-ID: <8299188.1156802363322.JavaMail.jira@brutus>
Date: Mon, 28 Aug 2006 14:59:23 -0700 (PDT)
From: "Michel Tourn (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Updated: (HADOOP-463) variable expansion in Configuration
In-Reply-To: <14071210.1155938956022.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org
X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N

     [ http://issues.apache.org/jira/browse/HADOOP-463?page=all ]

Michel Tourn updated HADOOP-463:
--------------------------------

    Attachment: confvar.patch

The patch implements the variable expansion described in this issue.
There is also a junit test.

On my shared client machine I use it like this.
(${user.name} expands to the System property "michel")

<property>
  <name>tmp.base</name>
  <value>/tmp/${user.name}</value>
</property>

<property>
  <name>dfs.name.dir</name>
  <value>${tmp.base}/hadoop/dfs/name</value>
</property>

etc.

> variable expansion in Configuration
> -----------------------------------
>
>                 Key: HADOOP-463
>                 URL: http://issues.apache.org/jira/browse/HADOOP-463
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Michel Tourn
>         Attachments: confvar.patch
>
>
> Add variable expansion to Configuration class.
> =================
> This is necessary for shared, client-side configurations:
> A Job submitter (an HDFS client) requires:
> <name>dfs.data.dir</name><value>/tmp/${user.name}/dfs</value>
> A local-mode mapreduce requires:
> <name>mapred.temp.dir</name><value>/tmp/${user.name}/mapred/tmp</value>
> Why this is necessary :
> =================
> Currently we use shared directories like:
> <name>dfs.data.dir</name><value>/tmp/dfs</value>
> This superficially seems to work.
> After all, different JVM clients create their own private subdirectory map_xxxx., so they will not conflict.
> What really happens:
> 1. /tmp/ is world-writable, as it's supposed to.
> 2. Hadoop will create missing subdirectories. 
> This is Java so that for ex. /tmp/system is created as writable only by the JVM process user
> 3. This is a shared client machine so next user's JVM will find /tmp/system owned by somebody else. Creating a directory within /tmp/system fails
> Implementation of var expansion
> =============
> in class Configuration, 
> The Properties really store things like put("banner", "hello ${user.name}");
> In public String get(String name): postprocess the returned value:
> Use a regexp to find the pattern ${xxxx}
> Lookup xxxx as a system property
> If found, replace ${xxxx} by the system property value.
> Else leave as-is. An unexpanded ${xxxx} is a hint that the variable name is invalid.
> Other workarounds 
> ===============
> The other proposed workarounds are not as elegant as variable expansion.
> Workaround 1: 
> have an installation script which does:
> mkdir /tmp/dfs
> chmod uga+rw /tmp/dfs
> repeat for ALL configured subdirectories at ANY nesting level
> keep the script in sync with changes to hadoop XML configuration files.
> Support the script on non-Unix platform
> Make sure the installtion script runs before Hadoop runs for the first time.
> If users change the permissions/delete any of the shared directories, it breaks again.
> Workaround 2: 
> do the chmod operations from within the Hadoop code.
> In pure java 1.4, 1.5 this is not possible.
> It requires the Hadoop client process to have chmod privilege (rather than just mkdir privilege)
> It requires to special-case directory creation code.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira