Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of forrie@gmail.com designates
 209.85.128.47 as permitted sender)
Message-ID: <52A4F28E.1010706@gmail.com>
Date: Sun, 08 Dec 2013 17:28:30 -0500
From: Forrest Aldrich <forrie@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9;
 rv:24.0) Gecko/20100101 Thunderbird/24.1.1
MIME-Version: 1.0
To: user@hadoop.apache.org
Subject: Re: Setting the HADOOP_HOME_DIR
References: <52A4C683.6010602@gmail.com>
 <CAHodO=JcD8CvYechLXX-UbsjvgGhK1tAkS7d_spe3mm8VmvqzQ@mail.gmail.com>
In-Reply-To: 
 <CAHodO=JcD8CvYechLXX-UbsjvgGhK1tAkS7d_spe3mm8VmvqzQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------080307070603030403060502"

This is a multi-part message in MIME format.
--------------080307070603030403060502
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Adam, my apologies, you are correct.  I meant to refer to the 
HADOOP_CONF_DIR -- this is what I get for posting before I've had enough 
coffee :-)

I want to put the CONF and LOGS directories outside of the /usr/hadoop 
structure -- this will make it easier to manage with Puppet, and prevent 
accidental overwrites during upgrades etc.


Thanks.


On 12/8/13, 4:56 PM, Adam Kawa wrote:
> Hi,
>
> If am not sure, if I understood your issue correctly. Would you like 
> to specify somehow where the configuration directory for your Hadoop 
> cluster is located (e.g. /etc/hadoop/conf)?
>
> If you use init scripts from CDH, they assume that config directory is 
> CONF_DIR="/etc/hadoop/conf".
> Afaik, when you use HDP or Apache distribution, then you can specify 
> where your configuration directory is when you start a script e.g. 
> "sudo -u hdfs /usr/lib/hadoop/sbin/hadoop-daemon.sh --config 
> <config_directory> start datanode"
>
> PS.
> I grepped my configuration directory, and installation directory 
> (/usr/lib/hadoop), but I can not see variable called: HADOOP_HOME_DIR 
> anywhere.
> I see that /usr/lib/hadoop/libexec/hadoop-layout.sh contains variable 
> HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf"}.
>
>
> 2013/12/8 Forrest Aldrich <forrie@gmail.com <mailto:forrie@gmail.com>>
>
>     We have a home-grown Hadoop installation that I am primarily
>     responsible for.   I've managed to get new nodes rolled out, it
>     takes time as we have other dependencies.   One item I've not been
>     able figure out is where to set the HADOOP_HOME_DIR variable, so I
>     can store the actual configuration for each node separate from the
>     binary tree.
>
>     Can anyone point me to where this gets set properly? We have an
>     init.d script that starts the services on the master, which calls
>     out to the slaves (as user "hadoop") -- but I'm guessing the
>     variable can be started there, exported and inherited -- but
>     perhaps it may be more proper to set in ~hadoop/conf/hadoop-env.sh.
>
>     The idea is to enable me to more easily roll out slaves, perhaps
>     using Puppet, so that the CONF and LOGS directories are separate
>     -- it's easier to manage that way.
>
>
>     Thanks.
>
>


--------------080307070603030403060502
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <font face="Arial">Adam, my apologies, you are correct.&nbsp; I meant to
      refer to the HADOOP_CONF_DIR -- this is what I get for posting
      before I've had enough coffee :-)<br>
      <br>
      I want to put the CONF and LOGS directories outside of the
      /usr/hadoop structure -- this will make it easier to manage with
      Puppet, and prevent accidental overwrites during upgrades etc.<br>
      <br>
      <br>
      Thanks.<br>
      <br>
      <br>
    </font>
    <div class="moz-cite-prefix">On 12/8/13, 4:56 PM, Adam Kawa wrote:<br>
    </div>
    <blockquote
cite="mid:CAHodO=JcD8CvYechLXX-UbsjvgGhK1tAkS7d_spe3mm8VmvqzQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>Hi,</div>
        <div><br>
        </div>
        <div>If am not sure, if I understood your issue correctly. Would
          you like to specify somehow where the configuration directory
          for your Hadoop cluster is located (e.g. /etc/hadoop/conf)?</div>
        <div><br>
        </div>
        <div>If you use init scripts from CDH, they assume that config
          directory is CONF_DIR="/etc/hadoop/conf".</div>
        <div>Afaik, when you use HDP or Apache distribution, then you
          can specify where your configuration directory is when you
          start a script e.g. "sudo -u hdfs
          /usr/lib/hadoop/sbin/hadoop-daemon.sh --config
          &lt;config_directory&gt; start datanode"</div>
        <div><br>
        </div>
        <div>PS.</div>
        <div><span style="font-family:Arial;font-size:13px">I grepped my
            configuration directory, and installation directory
            (/usr/lib/hadoop), but&nbsp;</span>I can not see variable
          called:&nbsp;<span style="font-family:Arial;font-size:13px">HADOOP_HOME_DIR
            anywhere.</span></div>
        <div>I see that&nbsp;/usr/lib/hadoop/libexec/hadoop-layout.sh
          contains variable
          HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf"}.</div>
      </div>
      <div class="gmail_extra"><br>
        <br>
        <div class="gmail_quote">2013/12/8 Forrest Aldrich <span
            dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:forrie@gmail.com" target="_blank">forrie@gmail.com</a>&gt;</span><br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000"> <font face="Arial">We
                have a home-grown Hadoop installation that I am
                primarily responsible for.&nbsp;&nbsp; I've managed to get new
                nodes rolled out, it takes time as we have other
                dependencies.&nbsp;&nbsp; One item I've not been able figure out
                is where to set the HADOOP_HOME_DIR variable, so I can
                store the actual configuration for each node separate
                from the binary tree.<br>
                <br>
                Can anyone point me to where this gets set properly?&nbsp;&nbsp;
                We have an init.d script that starts the services on the
                master, which calls out to the slaves (as user "hadoop")
                -- but I'm guessing the variable can be started there,
                exported and inherited -- but perhaps it may be more
                proper to set in ~hadoop/conf/hadoop-env.sh.<br>
                <br>
                The idea is to enable me to more easily roll out slaves,
                perhaps using Puppet, so that the CONF and LOGS
                directories are separate -- it's easier to manage that
                way.<br>
                <br>
                <br>
                Thanks.<br>
                <br>
              </font> </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>

--------------080307070603030403060502--