Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
To: user@hadoop.apache.org
From: Henning Blohm <henning.blohm@zfabrik.de>
Subject: Best way to pass configuration properties to MRv2 jobs
Message-ID: <571926B6.9030707@zfabrik.de>
Date: Thu, 21 Apr 2016 21:15:02 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.6.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

in our Hadoop 2.6.0 cluster, we need to pass some properties to all 
Hadoop processes so they can be referenced using ${...} syntax in 
configuration files. This works reasonably well using 
HADOOP_NAMENODE_OPTS and the like.

For Map/Reduce jobs however, we need to speficy not only

mapred.child.java.opts

to pass system properties, in addition we need to set

yarn.app.mapreduce.am.command-opts

for anything that is referenced in Hadoop configuration files.

In the end however almost all the properties passed are available as 
environment variables as well.

Hence my question:

* Is it possible to use reference environment variables in configuration 
files directly?
* Does anybody know of a simpler way to make sure some system properties 
are _always_ set for all Yarn processes?

Thanks,
Henning


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org