incubator-ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tapper, Gunnar" <gunnar.tap...@hp.com>
Subject RE: Modifying *-env.sh files using the Ambari REST API
Date Wed, 28 Aug 2013 18:23:38 GMT
HI Sid,

Thanks for the information. So, the tag has to be of the format "version<number>" rather
than the tag I specified?

I'm a bit confused about what you're saying about HADOOP_NAMENODE_OPTS. Are you saying that
it's not possible to add properties such as Dcom.sun.management.jmxremote.port to HADOOP_NAMENODE_OPTS
because the settings in the globals configuration file are used?

Sincerely,

Gunnar

I skate to where the puck is going to be, not where it has been.  - Wayne Gretzky


From: Siddharth Wagle [mailto:swagle@hortonworks.com]
Sent: Wednesday, August 28, 2013 12:15 PM
To: ambari-user@incubator.apache.org
Subject: Re: Modifying *-env.sh files using the Ambari REST API

Hi Gunnar,

"message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception
occurred: Configuration with that tag exists for 'hadoop-env.sh'"
Each new configuration needs to have a new tag, (version 1, 2 ... so on).
There is a configs script that will do this for you, check out configs.sh checked in under
"ambari-server/src/main/resources/scripts/", on your cluster it should be under /var/lib/ambari-server/resources/scripts.

Back to what you are trying to do, the HADOOP_NAMENODE_OPTS property is not substituted asis
from the configs on the ambari agent.
The HADOOP_NAMENODE_OPTS uses the following properties from the configuration named "global"
(http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/configrurations?type=global&tag=version1<http://%3cambari-server%3e:8080/api/v1/clusters/%3ccluster-name%3e/configrurations?type=global&tag=version1>)

namenode_opt_newsize
namenode_opt_maxnewsize
namenode_heapsize
hdfs_log_dir_prefix
You can change these properties in the global using the API or using the UI for HDFS service.
Best Regards,
Sid



On Wed, Aug 28, 2013 at 10:41 AM, Tapper, Gunnar <gunnar.tapper@hp.com<mailto:gunnar.tapper@hp.com>>
wrote:
Hi,

I need to update the Hadoop and HBase *-env.sh files programmatically to change; for example,
the HADOOP_NAMENODE_OPTS environmental variable. I do the following:

1.       Check content of /etc/hadoop/conf/hadoop-env.sh.

cat hadoop-env.sh | grep HADOOP_NAMENODE_OPTS
export HADOOP_NAMENODE_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log
-XX:NewSize=200m -XX:MaxNewSize=640m -Xloggc:/var/log/hadoop/$USER/gc.log-`date +'%Y%m%d%H%M'`
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms1024m -Xmx1024m
-Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT ${HADOOP_NAMENODE_OPTS}"

2.       Stop MapReduce and HDFS using the Ambari web client (I haven't figured out the REST
calls yet).

3.       Issue the REST API call, which generates an error.

curl -u admin:admin -i -X PUT -d '{"Clusters": {"desired_config": {"type": "hadoop-env.sh",
"tag": "seapilot-install", "properties" : { "HADOOP_NAMENODE_OPTS" : "-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=58004 $HADOOP_NAMENODE_OPTS"}}}}' http://<ambari-server>:8080/api/v1/clusters/mycluster<http://%3cambari-server%3e:8080/api/v1/clusters/mycluster>
HTTP/1.1 500 Server Error
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Server: Jetty(7.6.7.v20120910)
Cache-Control: proxy-revalidate
Content-Length: 188
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Set-Cookie: AMBARISESSIONID=1i1srtsmvh76f1688i8zcl5995;Path=/
Date: Wed, 28 Aug 2013 17:25:17 GMT

{
  "status" : 500,
  "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system
exception occurred: Configuration with that tag exists for 'hadoop-env.sh'"
}

4.       Check content of /etc/hadoop/conf/hadoop-env.sh.

cat hadoop-env.sh | grep HADOOP_NAMENODE_OPTS
export HADOOP_NAMENODE_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log
-XX:NewSize=200m -XX:MaxNewSize=640m -Xloggc:/var/log/hadoop/$USER/gc.log-`date +'%Y%m%d%H%M'`
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms1024m -Xmx1024m
-Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT ${HADOOP_NAMENODE_OPTS}"


On a Regular Hadoop installation (that is, downloaded from the Apache website), I simply add
a new line to hadoop-env.sh at the end thereby redefining the environment variable; for example:

export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
   -Dcom.sun.management.jmxremote.ssl=false
   -Dcom.sun.management.jmxremote.authenticate=false
   -Dcom.sun.management.jmxremote.port=58004
    $HADOOP_NAMENODE_OPTS"

I know that I can make the changes directly to the *-env.sh files and then start/stop the
services w/o using the Ambari GUI/REST calls but that defeats the purpose.

Can this configuration change be done in Ambari?

Sincerely,

Gunnar

I skate to where the puck is going to be, not where it has been. - Wayne Gretzky







CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.

Mime
View raw message