incubator-ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Wagle <swa...@hortonworks.com>
Subject Re: Modifying *-env.sh files using the Ambari REST API
Date Wed, 28 Aug 2013 21:41:04 GMT
Hi Gunnar,

The version tag can be anything, the web UI sets it version + timestamp.
Just that it needs to be unique for every PUT request that succeeds in
creating a configuration.
Look at,
https://github.com/apache/ambari/blob/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/hadoop-env.sh.erb.
(Line 43)

This property cannot be changed through configs as of now,
HADOOP_NAMENODE_OPTS

You might want to open a JIRA for this.

-Sid


On Wed, Aug 28, 2013 at 11:23 AM, Tapper, Gunnar <gunnar.tapper@hp.com>wrote:

>  HI Sid,****
>
> ** **
>
> Thanks for the information. So, the tag has to be of the format
> “version<number>” rather than the tag I specified?****
>
> ** **
>
> I’m a bit confused about what you’re saying about HADOOP_NAMENODE_OPTS.
> Are you saying that it’s not possible to add properties such as
> Dcom.sun.management.jmxremote.port to HADOOP_NAMENODE_OPTS because the
> settings in the globals configuration file are used?****
>
> ** **
>
> Sincerely,****
>
> ** **
>
> Gunnar****
>
> ** **
>
> *I skate to where the puck is going to be, not where it has been. * —
> Wayne Gretzky****
>
> ** **
>
> ** **
>
> *From:* Siddharth Wagle [mailto:swagle@hortonworks.com]
> *Sent:* Wednesday, August 28, 2013 12:15 PM
> *To:* ambari-user@incubator.apache.org
> *Subject:* Re: Modifying *-env.sh files using the Ambari REST API****
>
> ** **
>
> Hi Gunnar,
>
> "message" : "org.apache.ambari.server.controller.spi.SystemException: An
> internal system exception occurred: Configuration with that tag exists for
> 'hadoop-env.sh'"****
>
> Each new configuration needs to have a new tag, (version 1, 2 ... so on).*
> ***
>
> There is a configs script that will do this for you, check out configs.sh
> checked in under "ambari-server/src/main/resources/scripts/", on your
> cluster it should be under /var/lib/ambari-server/resources/scripts.****
>
> ** **
>
> Back to what you are trying to do, the HADOOP_NAMENODE_OPTS property is
> not substituted asis from the configs on the ambari agent. ****
>
> The HADOOP_NAMENODE_OPTS uses the following properties from the
> configuration named "global" (
> http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/configrurations?type=global&tag=version1
> )
>
> namenode_opt_newsize
> namenode_opt_maxnewsize
> namenode_heapsize
> hdfs_log_dir_prefix****
>
> You can change these properties in the global using the API or using the
> UI for HDFS service.****
>
> Best Regards,****
>
> Sid****
>
>
>  ****
>
> ** **
>
> On Wed, Aug 28, 2013 at 10:41 AM, Tapper, Gunnar <gunnar.tapper@hp.com>
> wrote:****
>
> Hi,****
>
>  ****
>
> I need to update the Hadoop and HBase *-env.sh files programmatically to
> change; for example, the HADOOP_NAMENODE_OPTS environmental variable. I do
> the following:****
>
>  ****
>
> **1.       **Check content of /etc/hadoop/conf/hadoop-env.sh.****
>
>  ****
>
> cat hadoop-env.sh | grep HADOOP_NAMENODE_OPTS****
>
> export HADOOP_NAMENODE_OPTS="-server -XX:ParallelGCThreads=8
> -XX:+UseConcMarkSweepGC
> -XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log -XX:NewSize=200m
> -XX:MaxNewSize=640m -Xloggc:/var/log/hadoop/$USER/gc.log-`date
> +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -XX:+PrintGCDateStamps -Xms1024m -Xmx1024m
> -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT
> ${HADOOP_NAMENODE_OPTS}"****
>
>  ****
>
> **2.       **Stop MapReduce and HDFS using the Ambari web client (I
> haven’t figured out the REST calls yet).****
>
>  ****
>
> **3.       **Issue the REST API call, which generates an error.****
>
>  ****
>
> curl -u admin:admin -i -X PUT -d '{"Clusters": {"desired_config": {"type":
> "hadoop-env.sh", "tag": "seapilot-install", "properties" : {
> "HADOOP_NAMENODE_OPTS" : "-Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.port=58004 $HADOOP_NAMENODE_OPTS"}}}}'
> http://<ambari-server>:8080/api/v1/clusters/mycluster****
>
> HTTP/1.1 500 Server Error****
>
> Expires: Thu, 01 Jan 1970 00:00:00 GMT****
>
> Content-Type: text/plain****
>
> Server: Jetty(7.6.7.v20120910)****
>
> Cache-Control: proxy-revalidate****
>
> Content-Length: 188****
>
> Proxy-Connection: Keep-Alive****
>
> Connection: Keep-Alive****
>
> Set-Cookie: AMBARISESSIONID=1i1srtsmvh76f1688i8zcl5995;Path=/****
>
> Date: Wed, 28 Aug 2013 17:25:17 GMT****
>
>  ****
>
> {****
>
>   "status" : 500,****
>
>   "message" : "org.apache.ambari.server.controller.spi.SystemException: An
> internal system exception occurred: Configuration with that tag exists for
> 'hadoop-env.sh'"****
>
> }****
>
>  ****
>
> **4.       **Check content of /etc/hadoop/conf/hadoop-env.sh.****
>
>  ****
>
> cat hadoop-env.sh | grep HADOOP_NAMENODE_OPTS****
>
> export HADOOP_NAMENODE_OPTS="-server -XX:ParallelGCThreads=8
> -XX:+UseConcMarkSweepGC
> -XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log -XX:NewSize=200m
> -XX:MaxNewSize=640m -Xloggc:/var/log/hadoop/$USER/gc.log-`date
> +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -XX:+PrintGCDateStamps -Xms1024m -Xmx1024m
> -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT
> ${HADOOP_NAMENODE_OPTS}"****
>
>  ****
>
>  ****
>
> On a Regular Hadoop installation (that is, downloaded from the Apache
> website), I simply add a new line to hadoop-env.sh at the end thereby
> redefining the environment variable; for example:****
>
>  ****
>
> export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote****
>
>    -Dcom.sun.management.jmxremote.ssl=false****
>
>    -Dcom.sun.management.jmxremote.authenticate=false****
>
>    -Dcom.sun.management.jmxremote.port=58004****
>
>     $HADOOP_NAMENODE_OPTS"****
>
>  ****
>
> I know that I can make the changes directly to the *-env.sh files and then
> start/stop the services w/o using the Ambari GUI/REST calls but that
> defeats the purpose. ****
>
>  ****
>
> Can this configuration change be done in Ambari? ****
>
>  ****
>
> Sincerely,****
>
>  ****
>
> Gunnar****
>
>  ****
>
> *I skate to where the puck is going to be, not where it has been. *—
> Wayne Gretzky****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> ** **
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.****
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message