Return-Path: X-Original-To: apmail-ambari-commits-archive@www.apache.org Delivered-To: apmail-ambari-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AD1791766C for ; Tue, 7 Oct 2014 22:52:32 +0000 (UTC) Received: (qmail 65527 invoked by uid 500); 7 Oct 2014 22:52:32 -0000 Delivered-To: apmail-ambari-commits-archive@ambari.apache.org Received: (qmail 65437 invoked by uid 500); 7 Oct 2014 22:52:32 -0000 Mailing-List: contact commits-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: ambari-dev@ambari.apache.org Delivered-To: mailing list commits@ambari.apache.org Received: (qmail 64615 invoked by uid 99); 7 Oct 2014 22:52:31 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Oct 2014 22:52:31 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id C4896905D36; Tue, 7 Oct 2014 22:52:31 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: yusaku@apache.org To: commits@ambari.apache.org Date: Tue, 07 Oct 2014 22:53:15 -0000 Message-Id: In-Reply-To: <478d9103dc2f4d60aae0f04f68dae6a1@git.apache.org> References: <478d9103dc2f4d60aae0f04f68dae6a1@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [47/51] [partial] AMBARI-7621. Import initial contribution for Ambari support on Windows to branch-windows-dev. (Jayush Luniya and Florian Barca via yusaku) http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-admin/src/main/resources/view.xml ---------------------------------------------------------------------- diff --git a/ambari-admin/src/main/resources/view.xml b/ambari-admin/src/main/resources/view.xml index 9bb4d8e..57d09e9 100644 --- a/ambari-admin/src/main/resources/view.xml +++ b/ambari-admin/src/main/resources/view.xml @@ -18,10 +18,7 @@ limitations under the License. Kerberos, LDAP, Custom. Binary/Htt ADMIN_VIEW Ambari Admin View 1.0.0 - true INSTANCE - false - This view provides administration capabilities for Ambari \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/conf/unix/ambari-agent.ini ---------------------------------------------------------------------- diff --git a/ambari-agent/conf/unix/ambari-agent.ini b/ambari-agent/conf/unix/ambari-agent.ini index 162041a..ac37f16 100644 --- a/ambari-agent/conf/unix/ambari-agent.ini +++ b/ambari-agent/conf/unix/ambari-agent.ini @@ -19,12 +19,10 @@ secured_url_port=8441 [agent] prefix=/var/lib/ambari-agent/data -tmp_dir=/var/lib/ambari-agent/data/tmp ;loglevel=(DEBUG/INFO) loglevel=INFO data_cleanup_interval=86400 data_cleanup_max_age=2592000 -data_cleanup_max_size_MB = 100 ping_port=8670 cache_dir=/var/lib/ambari-agent/cache tolerate_download_failures=true http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/conf/unix/install-helper.sh ---------------------------------------------------------------------- diff --git a/ambari-agent/conf/unix/install-helper.sh b/ambari-agent/conf/unix/install-helper.sh index 3f47a79..70cf6d0 100644 --- a/ambari-agent/conf/unix/install-helper.sh +++ b/ambari-agent/conf/unix/install-helper.sh @@ -19,13 +19,9 @@ ################################################################## COMMON_DIR="/usr/lib/python2.6/site-packages/ambari_commons" -RESOURCE_MANAGEMENT_DIR="/usr/lib/python2.6/site-packages/resource_management" -JINJA_DIR="/usr/lib/python2.6/site-packages/ambari_jinja2" OLD_COMMON_DIR="/usr/lib/python2.6/site-packages/common_functions" INSTALL_HELPER_SERVER="/var/lib/ambari-server/install-helper.sh" COMMON_DIR_AGENT="/usr/lib/ambari-agent/lib/ambari_commons" -RESOURCE_MANAGEMENT_DIR_AGENT="/usr/lib/ambari-agent/lib/resource_management" -JINJA_AGENT_DIR="/usr/lib/ambari-agent/lib/ambari_jinja2" PYTHON_WRAPER_TARGET="/usr/bin/ambari-python-wrap" PYTHON_WRAPER_SOURCE="/var/lib/ambari-agent/ambari-python-wrap" @@ -36,14 +32,6 @@ do_install(){ if [ ! -d "$COMMON_DIR" ]; then ln -s "$COMMON_DIR_AGENT" "$COMMON_DIR" fi - # setting resource_management shared resource - if [ ! -d "$RESOURCE_MANAGEMENT_DIR" ]; then - ln -s "$RESOURCE_MANAGEMENT_DIR_AGENT" "$RESOURCE_MANAGEMENT_DIR" - fi - # setting jinja2 shared resource - if [ ! -d "$JINJA_DIR" ]; then - ln -s "$JINJA_AGENT_DIR" "$JINJA_DIR" - fi # setting python-wrapper script if [ ! -f "$PYTHON_WRAPER_TARGET" ]; then ln -s "$PYTHON_WRAPER_SOURCE" "$PYTHON_WRAPER_TARGET" @@ -51,6 +39,9 @@ do_install(){ } do_remove(){ + if [ -d "$COMMON_DIR" ]; then # common dir exists + rm -f "$COMMON_DIR" + fi if [ -f "$PYTHON_WRAPER_TARGET" ]; then rm -f "$PYTHON_WRAPER_TARGET" http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/conf/windows/ambari-agent.cmd ---------------------------------------------------------------------- diff --git a/ambari-agent/conf/windows/ambari-agent.cmd b/ambari-agent/conf/windows/ambari-agent.cmd new file mode 100644 index 0000000..e159be4 --- /dev/null +++ b/ambari-agent/conf/windows/ambari-agent.cmd @@ -0,0 +1,18 @@ +@echo off +REM Licensed to the Apache Software Foundation (ASF) under one or more +REM contributor license agreements. See the NOTICE file distributed with +REM this work for additional information rega4rding copyright ownership. +REM The ASF licenses this file to You under the Apache License, Version 2.0 +REM (the "License"); you may not use this file except in compliance with +REM the License. You may obtain a copy of the License at +REM +REM http://www.apache.org/licenses/LICENSE-2.0 +REM +REM Unless required by applicable law or agreed to in writing, software +REM distributed under the License is distributed on an "AS IS" BASIS, +REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +REM See the License for the specific language governing permissions and +REM limitations under the License. + +call ambari-env.cmd +powershell -ExecutionPolicy unrestricted -File ambari-agent.ps1 %* http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/conf/windows/ambari-agent.ini ---------------------------------------------------------------------- diff --git a/ambari-agent/conf/windows/ambari-agent.ini b/ambari-agent/conf/windows/ambari-agent.ini new file mode 100644 index 0000000..68fee1f --- /dev/null +++ b/ambari-agent/conf/windows/ambari-agent.ini @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific + +[server] +hostname=localhost +url_port=8440 +secured_url_port=8441 + +[agent] +prefix=data +;loglevel=(DEBUG/INFO) +loglevel=INFO +data_cleanup_interval=86400 +data_cleanup_max_age=2592000 +ping_port=8670 +cache_dir=cache +tolerate_download_failures=true + +[command] +maxretries=2 +sleepBetweenRetries=1 + +[security] +keysdir=keys +server_crt=ca.crt +passphrase_env_var_name=AMBARI_PASSPHRASE + +[services] +pidLookupPath=\\var\\run\\ambari-agent + +[heartbeat] +state_interval=6 +dirs=/etc/hadoop,/etc/hadoop/conf,/etc/hbase,/etc/hcatalog,/etc/hive,/etc/oozie, + /etc/sqoop,/etc/ganglia,/etc/nagios, + /var/run/hadoop,/var/run/zookeeper,/var/run/hbase,/var/run/templeton,/var/run/oozie, + /var/log/hadoop,/var/log/zookeeper,/var/log/hbase,/var/run/templeton,/var/log/hive, + /var/log/nagios +rpms=nagios,ganglia, + hadoop,hadoop-lzo,hbase,oozie,sqoop,pig,zookeeper,hive,libconfuse,ambari-log4j +; 0 - unlimited +log_lines_count=300 http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/conf/windows/ambari-agent.ps1 ---------------------------------------------------------------------- diff --git a/ambari-agent/conf/windows/ambari-agent.ps1 b/ambari-agent/conf/windows/ambari-agent.ps1 new file mode 100644 index 0000000..97aa9c3 --- /dev/null +++ b/ambari-agent/conf/windows/ambari-agent.ps1 @@ -0,0 +1,256 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License + +# description: ambari-agent service +# processname: ambari-agent + +$VERSION="1.3.0-SNAPSHOT" +$HASH="testhash" + +switch ($($args[0])){ + "--version" { + echo "$VERSION" + exit 0 + } + "--hash" { + echo "$HASH" + exit 0 + } +} + +$AMBARI_AGENT="ambari-agent" +$AMBARI_SVC_NAME = "Ambari Agent" +$current_directory = (Get-Item -Path ".\" -Verbose).FullName +#environment variables used in python, check if they exists, otherwise set them to $current_directory +#and pass to child python process +$Env:PYTHONPATH="$current_directory\sbin;$($Env:PYTHONPATH)" +$Env:PYTHON = "python.exe" + +$AMBARI_LOG_DIR="\var\log\ambari-agent" +$OUTFILE_STDOUT=Join-Path -path $AMBARI_LOG_DIR -childpath "ambari-agent.stdout" +$OUTFILE_STDERR=Join-Path -path $AMBARI_LOG_DIR -childpath "ambari-agent.stderr" +$LOGFILE=Join-Path -path $AMBARI_LOG_DIR -childpath "ambari-agent.log" +$AMBARI_AGENT_PY_SCRIPT=Join-Path -path $PSScriptRoot -childpath "sbin\service_wrapper.py" +if($AMBARI_AGENT_PY_SCRIPT.Contains(' ')) +{ + $AMBARI_AGENT_PY_SCRIPT = """" + $AMBARI_AGENT_PY_SCRIPT + """" +} + +$OK=1 +$NOTOK=0 + + +# Reading the environment file +#if [ -a /var/lib/ambari-agent/ambari-env.sh ]; then +# . /var/lib/ambari-agent/ambari-env.sh +#fi + + +#echo $AMBARI_PASSPHRASE + +$retcode=0 + +function _exit($code) +{ + $host.SetShouldExit($code) + exit $code +} + +function _detect_python() +{ + if(![boolean]$(Get-Command $Env:PYTHON -ErrorAction SilentlyContinue)) + { + echo "ERROR: Can not find python.exe in PATH. Add python executable to PATH and try again." + _exit(1) + } +} + +function _echo([switch]$off) +{ + if($off) + { + try + { + stop-transcript|out-null + } + catch [System.InvalidOperationException] + {} + } + else + { + try + { + start-transcript|out-null + } + catch [System.InvalidOperationException] + {} + } +} + +#`@echo +#write-Host "Hello" +#write-Host "World" +#`@echo -off + +Function _pstart_brief($cmd_args) +{ + #start python with -u to make stdout and stderr unbuffered + $arguments = @("-u",$AMBARI_AGENT_PY_SCRIPT) + $cmd_args + + $psi = New-Object System.Diagnostics.ProcessStartInfo + + $psi.RedirectStandardError = $True + $psi.RedirectStandardOutput = $True + + $psi.UseShellExecute = $False + + $psi.FileName = $Env:PYTHON + $psi.Arguments = $arguments + #$psi.WindowStyle = WindowStyle.Hidden + + $process = [Diagnostics.Process]::Start($psi) + + $process.WaitForExit() + + Write-Output $process.StandardOutput.ReadToEnd() +} + +Function _start($cmd_args) +{ + echo "Starting $AMBARI_SVC_NAME..." + _echo -off + + _pstart_brief($cmd_args) + + $cnt = 0 + do + { + Start-Sleep -Milliseconds 250 + $svc = Get-Service -Name $AMBARI_SVC_NAME + $cnt += 1 + if ($cnt -eq 120) + { + echo "$AMBARI_SVC_NAME still starting...". + return + } + } + until($svc.status -eq "Running") + + echo "$AMBARI_SVC_NAME is running" +} + +Function _pstart($cmd_args) +{ + New-Item -ItemType Directory -Force -Path $AMBARI_LOG_DIR | Out-Null + + $arguments = @($AMBARI_AGENT_PY_SCRIPT) + $cmd_args + + $p = New-Object System.Diagnostics.Process + $p.StartInfo.UseShellExecute = $false + $p.StartInfo.FileName = $Env:PYTHON + $p.StartInfo.Arguments = $arguments + [void]$p.Start(); + + echo "Verifying $AMBARI_AGENT process status..." + if (!$p){ + echo "ERROR: $AMBARI_AGENT start failed" + $host.SetShouldExit(-1) + exit + } + echo "Agent log at: $LOGFILE" + + $p.WaitForExit() +} + +Function _pstart_ioredir($cmd_args) +{ + New-Item -ItemType Directory -Force -Path $AMBARI_LOG_DIR | Out-Null + + #start python with -u to make stdout and stderr unbuffered + $arguments = @("-u",$AMBARI_AGENT_PY_SCRIPT) + $cmd_args + $process = Start-Process -FilePath $Env:PYTHON -ArgumentList $arguments -WindowStyle Hidden -RedirectStandardError $OUTFILE_STDERR -RedirectStandardOutput $OUTFILE_STDOUT -PassThru + echo "Verifying $AMBARI_AGENT process status..." + if (!$process){ + echo "ERROR: $AMBARI_AGENT start failed" + $host.SetShouldExit(-1) + exit + } + echo "Agent stdout at: $OUTFILE_STDOUT" + echo "Agent stderr at: $OUTFILE_STDERR" + echo "Agent log at: $LOGFILE" + + $process.WaitForExit() +} + +Function _stop($cmd_args){ + echo "Stopping $AMBARI_SVC_NAME..." + + _pstart_brief($cmd_args) + + $cnt = 0 + do + { + Start-Sleep -Milliseconds 250 + $svc = Get-Service -Name $AMBARI_SVC_NAME + $cnt += 1 + if ($cnt -eq 40) + { + echo "$AMBARI_SVC_NAME still stopping...". + return + } + } + until($svc.status -eq "Stopped") + + echo "$AMBARI_SVC_NAME is stopped" +} + +Function _status($cmd_args){ + echo "Getting $AMBARI_SVC_NAME status..." + + _pstart_brief($cmd_args) +} + +# check for python before any action +_detect_python +switch ($($args[0])){ + "start" {_start $args} + "debug" + { + echo "Starting ambari-agent" + _pstart_ioredir $args + echo "Ambari Agent finished" + } + "stop" {_stop $args} + "restart" { + _stop @("stop") + _start @("start") + } + "status" {_status $args} + "setup" + { + echo "Installing ambari-agent" + _pstart $args + echo "Ambari Agent installation finished" + } + default { + echo "Usage: ambari-agent {start|stop|restart|setup|status} [options]" + echo "Use ambari-agent --help to get details on options available." + echo "Or, simply invoke ambari-agent.py --help to print the options." + $retcode=1 + } + } + +$host.SetShouldExit($retcode) +exit http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/conf/windows/ambari-env.cmd ---------------------------------------------------------------------- diff --git a/ambari-agent/conf/windows/ambari-env.cmd b/ambari-agent/conf/windows/ambari-env.cmd new file mode 100644 index 0000000..6e0c317 --- /dev/null +++ b/ambari-agent/conf/windows/ambari-env.cmd @@ -0,0 +1,22 @@ +@echo off +REM Licensed to the Apache Software Foundation (ASF) under one or more +REM contributor license agreements. See the NOTICE file distributed with +REM this work for additional information rega4rding copyright ownership. +REM The ASF licenses this file to You under the Apache License, Version 2.0 +REM (the "License"); you may not use this file except in compliance with +REM the License. You may obtain a copy of the License at +REM +REM http://www.apache.org/licenses/LICENSE-2.0 +REM +REM Unless required by applicable law or agreed to in writing, software +REM distributed under the License is distributed on an "AS IS" BASIS, +REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +REM See the License for the specific language governing permissions and +REM limitations under the License. + +set SERVICE_NAME=Ambari Agent +REM REM set AMBARI_AGENT_CONF_DIR=C:\work\test\agent +REM set AMBARI_AGENT_LOG_DIR=logs +set AGENT_SERVICE_WRAPPER=sbin\service_wrapper.py +REM python exe that will be used for command execution(must have access to pywin32 and agent python code) +set PYTHON_EXE=C:\Python27\python.exe http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/conf/windows/createservice.ps1 ---------------------------------------------------------------------- diff --git a/ambari-agent/conf/windows/createservice.ps1 b/ambari-agent/conf/windows/createservice.ps1 new file mode 100644 index 0000000..b19aef3 --- /dev/null +++ b/ambari-agent/conf/windows/createservice.ps1 @@ -0,0 +1,195 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License + +param( + [String] + [Parameter(Mandatory=$true )] + $username, + [String] + [Parameter(Mandatory=$true )] + $password, + [String] + [Parameter(Mandatory=$true )] + $servicename, + [String] + [Parameter(Mandatory=$true )] + $hdpResourcesDir, + [String] + [Parameter(Mandatory=$true )] + $servicecmdpath + ) + +function Invoke-Cmd ($command) +{ + Write-Output $command + $out = cmd.exe /C "$command" 2>&1 + Write-Output $out + return $out +} + +function Invoke-CmdChk ($command) +{ + Write-Output $command + $out = cmd.exe /C "$command" 2>&1 + Write-Output $out + if (-not ($LastExitCode -eq 0)) + { + throw "Command `"$out`" failed with exit code $LastExitCode " + } + return $out +} + +### Stops and deletes the Hadoop service. +function StopAndDeleteHadoopService( + [String] + [Parameter( Position=0, Mandatory=$true )] + $service +) +{ + Write-Output "Stopping $service" + $s = Get-Service $service -ErrorAction SilentlyContinue + + if( $s -ne $null ) + { + Stop-Service $service + $cmd = "sc.exe delete $service" + Invoke-Cmd $cmd + } +} + +# Convenience method for processing command-line credential objects +# Assumes $credentialsHash is a hash with one of the following being true: +# - keys "username" and "password"/"passwordBase64" are set to strings +# - key "credentialFilePath" is set to the path of a serialized PSCredential object +function Get-HadoopUserCredentials($credentialsHash) +{ + Write-Output "Using provided credentials for username $($credentialsHash["username"])" | Out-Null + $username = $credentialsHash["username"] + if($username -notlike "*\*") + { + $username = "$ENV:COMPUTERNAME\$username" + } + $securePassword = $credentialsHash["password"] | ConvertTo-SecureString -AsPlainText -Force + $creds = New-Object System.Management.Automation.PSCredential $username, $securePassword + return $creds +} + +### Creates and configures the service. +function CreateAndConfigureHadoopService( + [String] + [Parameter( Position=0, Mandatory=$true )] + $service, + [String] + [Parameter( Position=1, Mandatory=$true )] + $hdpResourcesDir, + [String] + [Parameter( Position=2, Mandatory=$true )] + $serviceBinDir, + [String] + [Parameter( Position=3, Mandatory=$true )] + $servicecmdpath, + [System.Management.Automation.PSCredential] + [Parameter( Position=4, Mandatory=$true )] + $serviceCredential +) +{ + if ( -not ( Get-Service "$service" -ErrorAction SilentlyContinue ) ) + { + Write-Output "Creating service `"$service`" as $serviceBinDir\$service.exe" + $xcopyServiceHost_cmd = "copy /Y `"$hdpResourcesDir\namenode.exe`" `"$serviceBinDir\$service.exe`"" + Invoke-CmdChk $xcopyServiceHost_cmd + + #HadoopServiceHost.exe will write to this log but does not create it + #Creating the event log needs to be done from an elevated process, so we do it here + if( -not ([Diagnostics.EventLog]::SourceExists( "$service" ))) + { + [Diagnostics.EventLog]::CreateEventSource( "$service", "" ) + } + Write-Output "Adding service $service" + if ($serviceCredential.Password.get_Length() -ne 0) + { + $s = New-Service -Name "$service" -BinaryPathName "$serviceBinDir\$service.exe" -Credential $serviceCredential -DisplayName "Apache Hadoop $service" + if ( $s -eq $null ) + { + throw "CreateAndConfigureHadoopService: Service `"$service`" creation failed" + } + } + else + { + # Separately handle case when password is not provided + # this path is used for creating services that run under (AD) Managed Service Account + # for them password is not provided and in that case service cannot be created using New-Service commandlet + $serviceUserName = $serviceCredential.UserName + $cred = $serviceCredential.UserName.Split("\") + + # Throw exception if domain is not specified + if (($cred.Length -lt 2) -or ($cred[0] -eq ".")) + { + throw "Environment is not AD or domain is not specified" + } + + $cmd="$ENV:WINDIR\system32\sc.exe create `"$service`" binPath= `"$serviceBinDir\$service.exe`" obj= $serviceUserName DisplayName= `"Apache Hadoop $service`" " + try + { + Invoke-CmdChk $cmd + } + catch + { + throw "CreateAndConfigureHadoopService: Service `"$service`" creation failed" + } + } + + $cmd="$ENV:WINDIR\system32\sc.exe failure $service reset= 30 actions= restart/5000" + Invoke-CmdChk $cmd + + $cmd="$ENV:WINDIR\system32\sc.exe config $service start= demand" + Invoke-CmdChk $cmd + + + Write-Output "Creating service config ${serviceBinDir}\$service.xml" + $cmd = "$servicecmdpath --service $service > `"$serviceBinDir\$service.xml`"" + Invoke-CmdChk $cmd + } + else + { + Write-Output "Service `"$service`" already exists, Removing `"$service`"" + StopAndDeleteHadoopService $service + CreateAndConfigureHadoopService $service $hdpResourcesDir $serviceBinDir $servicecmdpath $serviceCredential + } +} + + +try +{ + Write-Output "Creating credential object" + ### + ### Create the Credential object from the given username and password or the provided credentials file + ### + $serviceCredential = Get-HadoopUserCredentials -credentialsHash @{"username" = $username; "password" = $password} + $username = $serviceCredential.UserName + Write-Output "Username: $username" + + Write-Output "Creating service $service" + ### + ### Create Service + ### + CreateAndConfigureHadoopService $servicename $hdpResourcesDir $hdpResourcesDir $servicecmdpath $serviceCredential + Write-Output "Done" +} +catch +{ + Write-Output "Failure" + exit 1 +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/conf/windows/service_wrapper.py ---------------------------------------------------------------------- diff --git a/ambari-agent/conf/windows/service_wrapper.py b/ambari-agent/conf/windows/service_wrapper.py new file mode 100644 index 0000000..422106d --- /dev/null +++ b/ambari-agent/conf/windows/service_wrapper.py @@ -0,0 +1,219 @@ +''' +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +''' +import os +import optparse +import sys + +import win32serviceutil +import win32api +import win32event +import win32service + +from ambari_commons.ambari_service import AmbariService +from ambari_commons.exceptions import * +from ambari_commons.logging import * +from ambari_commons.os_windows import WinServiceController +from ambari_agent.AmbariConfig import * +from ambari_agent.HeartbeatHandlers_windows import HeartbeatStopHandler + +AMBARI_VERSION_VAR = "AMBARI_VERSION_VAR" + +ENV_PYTHONPATH = "PYTHONPATH" + +def parse_options(): + # parse env cmd + with open(os.path.join(os.getcwd(), "ambari-env.cmd"), "r") as env_cmd: + content = env_cmd.readlines() + for line in content: + if line.startswith("set"): + name, value = line[4:].split("=") + os.environ[name] = value.rstrip() + # checking env variables, and fallback to working dir if no env var was founded + if not os.environ.has_key("AMBARI_AGENT_CONF_DIR"): + os.environ["AMBARI_AGENT_CONF_DIR"] = os.getcwd() + if not os.environ.has_key("AMBARI_AGENT_LOG_DIR"): + os.environ["AMBARI_AGENT_LOG_DIR"] = os.path.join("\\", "var", "log", "ambari-agent") + if not os.path.exists(os.environ["AMBARI_AGENT_LOG_DIR"]): + os.makedirs(os.environ["AMBARI_AGENT_LOG_DIR"]) + + +class AmbariAgentService(AmbariService): + AmbariService._svc_name_ = "Ambari Agent" + AmbariService._svc_display_name_ = "Ambari Agent" + AmbariService._svc_description_ = "Ambari Agent" + + AmbariService._AdjustServiceVersion() + + heartbeat_stop_handler = None + + def SvcDoRun(self): + parse_options() + self.redirect_output_streams() + + # Soft dependency on the Windows Time service + ensure_time_service_is_started() + + self.heartbeat_stop_handler = HeartbeatStopHandler(self._heventSvcStop) + + self.ReportServiceStatus(win32service.SERVICE_RUNNING) + + from ambari_agent import main + main.main(self.heartbeat_stop_handler) + + def _InitOptionsParser(self): + return init_options_parser() + + def redirect_output_streams(self): + self._RedirectOutputStreamsToFile(AmbariConfig.getOutFile()) + pass + + +def ensure_time_service_is_started(): + ret = WinServiceController.EnsureServiceIsStarted("W32Time") + if 0 != ret: + raise FatalException(-1, "Error starting Windows Time service: " + string(ret)) + pass + + +def ctrlHandler(ctrlType): + AmbariAgentService.DefCtrlCHandler() + return True + +def svcsetup(): + AmbariAgentService.set_ctrl_c_handler(ctrlHandler) + AmbariAgentService.Install() + pass + +# +# Starts the Ambari Agent as a service. +# Start the Agent in normal mode, as a Windows service. If the Ambari Agent is +# not registered as a service, the function fails. By default, only one instance of the service can +# possibly run. +# +def svcstart(options): + if 0 != AmbariAgentService.Start(15): + options.exit_message = None + pass + +# +# Stops the Ambari Agent. +# +def svcstop(options): + if 0 != AmbariAgentService.Stop(): + options.exit_message = None + +# +# The Ambari Agent status. +# +def svcstatus(options): + options.exit_message = None + + statusStr = AmbariAgentService.QueryStatus() + print "Ambari Agent is " + statusStr + +def svcdebug(options): + sys.frozen = 'windows_exe' # Fake py2exe so we can debug + + AmbariAgentService.set_ctrl_c_handler(ctrlHandler) + win32serviceutil.HandleCommandLine(AmbariAgentService, options) + + +def init_options_parser(): + parser = optparse.OptionParser(usage="usage: %prog action [options]", ) + parser.add_option('-r', '--hostname', dest="host_name", default="localhost", + help="Use specified Ambari server host for registration.") + parser.add_option('-j', '--java-home', dest="java_home", default=None, + help="Use specified java_home. Must be valid on all hosts") + parser.add_option("-v", "--verbose", + action="store_true", dest="verbose", default=False, + help="Print verbose status messages") + parser.add_option("-s", "--silent", + action="store_true", dest="silent", default=False, + help="Silently accepts default prompt values") + parser.add_option('--jdbc-driver', default=None, + help="Specifies the path to the JDBC driver JAR file for the " \ + "database type specified with the --jdbc-db option. Used only with --jdbc-db option.", + dest="jdbc_driver") + return parser + + +# +# Main. +# +def agent_main(): + parser = init_options_parser() + (options, args) = parser.parse_args() + + options.warnings = [] + + if len(args) == 0: + print parser.print_help() + parser.error("No action entered") + + action = args[0] + possible_args_numbers = [1] + + matches = 0 + for args_number_required in possible_args_numbers: + matches += int(len(args) == args_number_required) + + if matches == 0: + print parser.print_help() + possible_args = ' or '.join(str(x) for x in possible_args_numbers) + parser.error("Invalid number of arguments. Entered: " + str(len(args)) + ", required: " + possible_args) + + options.exit_message = "Ambari Agent '%s' completed successfully." % action + try: + if action == SETUP_ACTION: + #TODO Insert setup(options) here upon need + svcsetup() + elif action == START_ACTION: + svcstart(options) + elif action == DEBUG_ACTION: + svcdebug(options) + elif action == STOP_ACTION: + svcstop(options) + elif action == STATUS_ACTION: + svcstatus(options) + else: + parser.error("Invalid action") + + if options.warnings: + for warning in options.warnings: + print_warning_msg(warning) + pass + options.exit_message = "Ambari Agent '%s' completed with warnings." % action + pass + except FatalException as e: + if e.reason is not None: + print_error_msg("Exiting with exit code {0}. \nREASON: {1}".format(e.code, e.reason)) + sys.exit(e.code) + except NonFatalException as e: + options.exit_message = "Ambari Agent '%s' completed with warnings." % action + if e.reason is not None: + print_warning_msg(e.reason) + + if options.exit_message is not None: + print options.exit_message + +if __name__ == '__main__': + try: + agent_main() + except (KeyboardInterrupt, EOFError): + print("\nAborting ... Keyboard Interrupt.") + sys.exit(1) http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/pom.xml ---------------------------------------------------------------------- diff --git a/ambari-agent/pom.xml b/ambari-agent/pom.xml index 3761649..71be41f 100644 --- a/ambari-agent/pom.xml +++ b/ambari-agent/pom.xml @@ -38,9 +38,9 @@ /var/run/ambari-agent false /usr/lib/python2.6/site-packages/ambari_agent + /usr/lib/python2.6/site-packages/resource_management /usr/lib/ambari-agent/lib/ambari_commons - /usr/lib/ambari-agent/lib/resource_management - /usr/lib/ambari-agent/lib/ambari_jinja2 + /usr/lib/python2.6/site-packages/jinja2 /usr/lib/ambari-agent/lib python >= 2.6 python (>= 2.6) @@ -49,7 +49,6 @@ ../ambari-server ${project.build.directory}/cache/ ${ambari.server.module}/src/main/python/ambari_server/resourceFilesKeeper.py - /etc/rc.d/init.d @@ -109,13 +108,13 @@ - ${project.basedir}/../ambari-common/src/main/unix/ambari-python-wrap + ${executable.python} src/test/python unitTests.py - ${project.basedir}/../ambari-common/src/main/python/ambari_jinja2:${project.basedir}/../ambari-common/src/main/python/ambari_commons:${project.basedir}/../ambari-common/src/main/python/resource_management:${project.basedir}/../ambari-common/src/test/python:${project.basedir}/../ambari-common/src/main/python:${project.basedir}/src/main/python/ambari_agent:${project.basedir}/src/test/python/ambari_agent:${project.basedir}/src/test/python/resource_management:${project.basedir}/src/main/python:${project.basedir}/../ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/package/files:${project.basedir}/../ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HDFS/package/files:$PYTHONPATH + ${path.python.1}${pathsep}$PYTHONPATH ${skipTests} @@ -128,15 +127,15 @@ - ${project.basedir}/../ambari-common/src/main/unix/ambari-python-wrap - target/ambari-agent-${project.version} + ${executable.python} + target${dirsep}ambari-agent-${project.version} - ${project.basedir}/src/main/python/setup.py + ${project.basedir}${dirsep}src${dirsep}main${dirsep}python${dirsep}setup.py clean bdist_dumb - target/ambari-agent-${project.version}:$PYTHONPATH + target${dirsep}ambari-agent-${project.version}${pathsep}$PYTHONPATH python-package @@ -147,14 +146,14 @@ - ${project.basedir}/../ambari-common/src/main/unix/ambari-python-wrap + ${executable.python} ${basedir} ${resource.keeper.script} ${target.cache.dir} - target/ambari-agent-${project.version}:$PYTHONPATH + target${dirsep}ambari-agent-${project.version}${pathsep}$PYTHONPATH generate-hash-files @@ -199,10 +198,6 @@ src/main/package/rpm/preremove.sh utf-8 - - src/main/package/rpm/posttrans_agent.sh - utf-8 - x86_64 false @@ -213,7 +208,7 @@ root - ${project.build.directory}/${project.artifactId}-${project.version}/ambari_agent + ${project.build.directory}${dirsep}${project.artifactId}-${project.version}${dirsep}ambari_agent @@ -230,22 +225,22 @@ - ${ambari_commons.install.dir} + ${resmgmt.install.dir} root root - ${project.basedir}/../ambari-common/src/main/python/ambari_commons + ${project.build.directory}/${project.artifactId}-${project.version}/resource_management - ${resource_management.install.dir} + ${ambari_commons.install.dir} root root - ${project.basedir}/../ambari-common/src/main/python/resource_management + ${project.basedir}/../ambari-common/src/main/python/ambari_commons @@ -255,9 +250,9 @@ root - ${project.basedir}/../ambari-common/src/main/python/ambari_jinja2/ambari_jinja2 + ${project.basedir}/../ambari-common/src/main/python/jinja2/jinja2 - ${project.basedir}/../ambari-common/src/main/python/ambari_jinja2/ambari_jinja2/testsuite + ${project.basedir}/../ambari-common/src/main/python/jinja2/jinja2/testsuite @@ -332,12 +327,6 @@ root - /var/lib/${project.artifactId}/data/tmp - 755 - root - root - - /var/lib/${project.artifactId}/keys 755 root @@ -350,7 +339,7 @@ root - ${init.d.dir} + /etc/rc.d/init.d 755 root root @@ -413,8 +402,18 @@ - ${project.basedir}/../ambari-common/src/main/python/ambari_jinja2/ambari_jinja2 - ${project.basedir}/../ambari-common/src/main/python/ambari_jinja2/ambari_jinja2/testsuite + ${project.build.directory}/${project.artifactId}-${project.version}/resource_management + directory + + perm + ${resmgmt.install.dir} + root + root + + + + ${project.basedir}/../ambari-common/src/main/python/jinja2/jinja2 + ${project.basedir}/../ambari-common/src/main/python/jinja2/jinja2/testsuite directory perm @@ -483,7 +482,6 @@ ${package.pid.dir} /var/lib/${project.artifactId}/data - /var/lib/${project.artifactId}/data/tmp /var/lib/${project.artifactId}/keys ${package.log.dir} @@ -551,19 +549,6 @@ root - - - ${project.basedir}/../ambari-common/src/main/python/resource_management - - directory - - perm - ${resource_management.install.dir} - 755 - root - root - - @@ -583,8 +568,7 @@ ${ambari.server.module}/src/main/resources - stacks/stack_advisor.py - stacks/${stack.distribution}/**/* + stacks/${hdpstackname}/**/* custom_actions/**/* false @@ -633,6 +617,7 @@ **/*.erb **/*.json **/*.pydevproject + **/*.wxs @@ -670,10 +655,151 @@ - suse11 - - /etc/init.d - + windows + + + win + + + + win + \ + ; + HDPWIN + python + cmd + cmd + .cmd + ${project.basedir}\..\ambari-common\src\main\python\jinja2;${project.basedir}\..\ambari-common\src\main\python\common_functions;${project.basedir}\..\ambari-common\src\test\python;${project.basedir}\..\ambari-common\src\main\python;${project.basedir}\src\main\python\ambari_agent;${project.basedir}\src\main\python\resource_management;${project.basedir}\src\test\python\ambari_agent;${project.basedir}\src\test\python\resource_management;${project.basedir}\src\main\python;${project.basedir}\..\ambari-server\src\main\resources\stacks\HDPWIN\2.1\services\HDFS\package\files + + + + linux + + + unix + + + + linux + / + : + HDP + ${project.basedir}/../ambari-common/src/main/unix/ambari-python-wrap + sh + sh + + ${project.basedir}/../ambari-common/src/main/python/jinja2:${project.basedir}/../ambari-common/src/main/python/common_functions:${project.basedir}/../ambari-common/src/test/python:${project.basedir}/../ambari-common/src/main/python:${project.basedir}/src/main/python/ambari_agent:${project.basedir}/src/main/python/resource_management:${project.basedir}/src/test/python/ambari_agent:${project.basedir}/src/test/python/resource_management:${project.basedir}/src/main/python:${project.basedir}/../ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/package/files:${project.basedir}/../ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HDFS/package/files + + + + windows-distro + + + Windows + + + + + + maven-assembly-plugin + + gnu + + src/packages/tarball/all.xml + src/packages/windows.xml + + + + + build-windows-zip + prepare-package + + single + + + + + + + org.codehaus.mojo + exec-maven-plugin + 1.2 + + + run-heat + package + + exec + + + heat.exe + + dir + "." + -dr + "AMBARI_AGENT_1.3.0_SNAPSHOT" + -platform + Win64 + -cg + "AmbariAgentGroup" + -gg + -ke + -srd + -o + ".\..\ambari-agent-files.wxs" + + ${basedir}/target/${final.name}-windows-dist + + + + + + org.apache.npanday.plugins + wix-maven-plugin + 1.4.0-incubating + true + + + src/main/package/msi/ambari-agent.wxs + target/ambari-agent-files.wxs + + target + + target/ambari-agent.wixobj + target/ambari-agent-files.wixobj + + target/ambari-agent-${project.version}.msi + + WixUIExtension + + + + + wix-candle + package + + candle + + + -arch x64 + + + + wix-light + package + + light + + + -b ${basedir}/target/${final.name}-windows-dist + + + + + + + http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/src/main/package/msi/ambari-agent.wxs ---------------------------------------------------------------------- diff --git a/ambari-agent/src/main/package/msi/ambari-agent.wxs b/ambari-agent/src/main/package/msi/ambari-agent.wxs new file mode 100644 index 0000000..cc2b8d8 --- /dev/null +++ b/ambari-agent/src/main/package/msi/ambari-agent.wxs @@ -0,0 +1,38 @@ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/src/main/package/rpm/posttrans_agent.sh ---------------------------------------------------------------------- diff --git a/ambari-agent/src/main/package/rpm/posttrans_agent.sh b/ambari-agent/src/main/package/rpm/posttrans_agent.sh deleted file mode 100644 index c301fc3..0000000 --- a/ambari-agent/src/main/package/rpm/posttrans_agent.sh +++ /dev/null @@ -1,38 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License - - -RESOURCE_MANAGEMENT_DIR="/usr/lib/python2.6/site-packages/resource_management" -RESOURCE_MANAGEMENT_DIR_AGENT="/usr/lib/ambari-agent/lib/resource_management" -JINJA_DIR="/usr/lib/python2.6/site-packages/ambari_jinja2" -JINJA_AGENT_DIR="/usr/lib/ambari-agent/lib/ambari_jinja2" - -# remove RESOURCE_MANAGEMENT_DIR if it's a directory -if [ -d "$RESOURCE_MANAGEMENT_DIR" ]; then # resource_management dir exists - if [ ! -L "$RESOURCE_MANAGEMENT_DIR" ]; then # resource_management dir is not link - rm -rf "$RESOURCE_MANAGEMENT_DIR" - fi -fi -# setting resource_management shared resource -if [ ! -d "$RESOURCE_MANAGEMENT_DIR" ]; then - ln -s "$RESOURCE_MANAGEMENT_DIR_AGENT" "$RESOURCE_MANAGEMENT_DIR" -fi - -# setting jinja2 shared resource -if [ ! -d "$JINJA_DIR" ]; then - ln -s "$JINJA_AGENT_DIR" "$JINJA_DIR" -fi - -exit 0 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/src/main/package/rpm/preremove.sh ---------------------------------------------------------------------- diff --git a/ambari-agent/src/main/package/rpm/preremove.sh b/ambari-agent/src/main/package/rpm/preremove.sh index 2078acd..5fb9be2 100644 --- a/ambari-agent/src/main/package/rpm/preremove.sh +++ b/ambari-agent/src/main/package/rpm/preremove.sh @@ -29,7 +29,7 @@ if [ "$1" -eq 0 ]; then # Action is uninstall /var/lib/ambari-agent/install-helper.sh remove fi - chkconfig --list | grep ambari-server && chkconfig --del ambari-server + chkconfig --del ambari-agent fi exit 0 http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/src/main/python/ambari_agent/ActionQueue.py ---------------------------------------------------------------------- diff --git a/ambari-agent/src/main/python/ambari_agent/ActionQueue.py b/ambari-agent/src/main/python/ambari_agent/ActionQueue.py index 094975d..5b09895 100644 --- a/ambari-agent/src/main/python/ambari_agent/ActionQueue.py +++ b/ambari-agent/src/main/python/ambari_agent/ActionQueue.py @@ -26,13 +26,11 @@ import pprint import os import json -from AgentException import AgentException from LiveStatus import LiveStatus from shell import shellRunner from ActualConfigHandler import ActualConfigHandler from CommandStatusDict import CommandStatusDict from CustomServiceOrchestrator import CustomServiceOrchestrator -from ambari_agent.BackgroundCommandExecutionHandle import BackgroundCommandExecutionHandle logger = logging.getLogger() @@ -54,7 +52,6 @@ class ActionQueue(threading.Thread): STATUS_COMMAND = 'STATUS_COMMAND' EXECUTION_COMMAND = 'EXECUTION_COMMAND' - BACKGROUND_EXECUTION_COMMAND = 'BACKGROUND_EXECUTION_COMMAND' ROLE_COMMAND_INSTALL = 'INSTALL' ROLE_COMMAND_START = 'START' ROLE_COMMAND_STOP = 'STOP' @@ -69,7 +66,6 @@ class ActionQueue(threading.Thread): super(ActionQueue, self).__init__() self.commandQueue = Queue.Queue() self.statusCommandQueue = Queue.Queue() - self.backgroundCommandQueue = Queue.Queue() self.commandStatuses = CommandStatusDict(callback_action = self.status_update_callback) self.config = config @@ -78,7 +74,8 @@ class ActionQueue(threading.Thread): self.configTags = {} self._stop = threading.Event() self.tmpdir = config.get('agent', 'prefix') - self.customServiceOrchestrator = CustomServiceOrchestrator(config, controller) + self.customServiceOrchestrator = CustomServiceOrchestrator(config, + controller) def stop(self): @@ -109,78 +106,32 @@ class ActionQueue(threading.Thread): command['serviceName'] + " of cluster " + \ command['clusterName'] + " to the queue.") logger.debug(pprint.pformat(command)) - if command['commandType'] == self.BACKGROUND_EXECUTION_COMMAND : - self.backgroundCommandQueue.put(self.createCommandHandle(command)) - else: - self.commandQueue.put(command) - - def cancel(self, commands): - for command in commands: - - logger.info("Canceling command {tid}".format(tid = str(command['target_task_id']))) - logger.debug(pprint.pformat(command)) - - task_id = command['target_task_id'] - reason = command['reason'] - - # Remove from the command queue by task_id - queue = self.commandQueue - self.commandQueue = Queue.Queue() - - while not queue.empty(): - queued_command = queue.get(False) - if queued_command['task_id'] != task_id: - self.commandQueue.put(queued_command) - else: - logger.info("Canceling " + queued_command['commandType'] + \ - " for service " + queued_command['serviceName'] + \ - " of cluster " + queued_command['clusterName'] + \ - " to the queue.") - - # Kill if in progress - self.customServiceOrchestrator.cancel_command(task_id, reason) + self.commandQueue.put(command) def run(self): while not self.stopped(): - self.processBackgroundQueueSafeEmpty(); - self.processStatusCommandQueueSafeEmpty(); - try: - command = self.commandQueue.get(True, self.EXECUTION_COMMAND_WAIT_TIME) - self.process_command(command) - except (Queue.Empty): - pass - def processBackgroundQueueSafeEmpty(self): - while not self.backgroundCommandQueue.empty(): - try: - command = self.backgroundCommandQueue.get(False) - if(command.has_key('__handle') and command['__handle'].status == None): + while not self.statusCommandQueue.empty(): + try: + command = self.statusCommandQueue.get(False) self.process_command(command) - except (Queue.Empty): - pass - - def processStatusCommandQueueSafeEmpty(self): - while not self.statusCommandQueue.empty(): + except (Queue.Empty): + pass try: - command = self.statusCommandQueue.get(False) + command = self.commandQueue.get(True, self.EXECUTION_COMMAND_WAIT_TIME) self.process_command(command) except (Queue.Empty): pass - def createCommandHandle(self, command): - if(command.has_key('__handle')): - raise AgentException("Command already has __handle") - command['__handle'] = BackgroundCommandExecutionHandle(command, command['commandId'], None, self.on_background_command_complete_callback) - return command + def process_command(self, command): logger.debug("Took an element of Queue: " + pprint.pformat(command)) # make sure we log failures - commandType = command['commandType'] try: - if commandType in [self.EXECUTION_COMMAND, self.BACKGROUND_EXECUTION_COMMAND]: + if command['commandType'] == self.EXECUTION_COMMAND: self.execute_command(command) - elif commandType == self.STATUS_COMMAND: + elif command['commandType'] == self.STATUS_COMMAND: self.execute_status_command(command) else: logger.error("Unrecognized command " + pprint.pformat(command)) @@ -191,11 +142,11 @@ class ActionQueue(threading.Thread): def execute_command(self, command): ''' - Executes commands of type EXECUTION_COMMAND + Executes commands of type EXECUTION_COMMAND ''' clusterName = command['clusterName'] commandId = command['commandId'] - isCommandBackground = command['commandType'] == self.BACKGROUND_EXECUTION_COMMAND + message = "Executing command with id = {commandId} for role = {role} of " \ "cluster {cluster}.".format( commandId = str(commandId), role=command['role'], @@ -206,8 +157,6 @@ class ActionQueue(threading.Thread): taskId = command['taskId'] # Preparing 'IN_PROGRESS' report in_progress_status = self.commandStatuses.generate_report_template(command) - # The path of the files that contain the output log and error log use a prefix that the agent advertises to the - # server. The prefix is defined in agent-config.ini in_progress_status.update({ 'tmpout': self.tmpdir + os.sep + 'output-' + str(taskId) + '.txt', 'tmperr': self.tmpdir + os.sep + 'errors-' + str(taskId) + '.txt', @@ -215,17 +164,13 @@ class ActionQueue(threading.Thread): 'status': self.IN_PROGRESS_STATUS }) self.commandStatuses.put_command_status(command, in_progress_status) - # running command commandresult = self.customServiceOrchestrator.runCommand(command, in_progress_status['tmpout'], in_progress_status['tmperr']) - - # dumping results - if isCommandBackground: - return - else: - status = self.COMPLETED_STATUS if commandresult['exitcode'] == 0 else self.FAILED_STATUS + status = self.COMPLETED_STATUS + if commandresult['exitcode'] != 0: + status = self.FAILED_STATUS roleResult = self.commandStatuses.generate_report_template(command) roleResult.update({ 'stdout': commandresult['stdout'], @@ -250,18 +195,6 @@ class ActionQueue(threading.Thread): # let ambari know that configuration tags were applied if status == self.COMPLETED_STATUS: configHandler = ActualConfigHandler(self.config, self.configTags) - #update - if command.has_key('forceRefreshConfigTags') and len(command['forceRefreshConfigTags']) > 0 : - - forceRefreshConfigTags = command['forceRefreshConfigTags'] - logger.info("Got refresh additional component tags command") - - for configTag in forceRefreshConfigTags : - configHandler.update_component_tag(command['role'], configTag, command['configurationTags'][configTag]) - - roleResult['customCommand'] = self.CUSTOM_COMMAND_RESTART # force restart for component to evict stale_config on server side - command['configurationTags'] = configHandler.read_actual_component(command['role']) - if command.has_key('configurationTags'): configHandler.write_actual(command['configurationTags']) roleResult['configurationTags'] = command['configurationTags'] @@ -274,40 +207,11 @@ class ActionQueue(threading.Thread): command['hostLevelParams'].has_key('custom_command') and \ command['hostLevelParams']['custom_command'] == self.CUSTOM_COMMAND_RESTART)): configHandler.write_actual_component(command['role'], command['configurationTags']) - if command['hostLevelParams'].has_key('clientsToUpdateConfigs') and \ - command['hostLevelParams']['clientsToUpdateConfigs']: - configHandler.write_client_components(command['serviceName'], command['configurationTags'], - command['hostLevelParams']['clientsToUpdateConfigs']) + configHandler.write_client_components(command['serviceName'], command['configurationTags']) roleResult['configurationTags'] = configHandler.read_actual_component(command['role']) self.commandStatuses.put_command_status(command, roleResult) - def command_was_canceled(self): - self.customServiceOrchestrator - def on_background_command_complete_callback(self, process_condenced_result, handle): - logger.debug('Start callback: %s' % process_condenced_result) - logger.debug('The handle is: %s' % handle) - status = self.COMPLETED_STATUS if handle.exitCode == 0 else self.FAILED_STATUS - - aborted_postfix = self.customServiceOrchestrator.command_canceled_reason(handle.command['taskId']) - if aborted_postfix: - status = self.FAILED_STATUS - logger.debug('Set status to: %s , reason = %s' % (status, aborted_postfix)) - else: - aborted_postfix = '' - - - roleResult = self.commandStatuses.generate_report_template(handle.command) - - roleResult.update({ - 'stdout': process_condenced_result['stdout'] + aborted_postfix, - 'stderr': process_condenced_result['stderr'] + aborted_postfix, - 'exitCode': process_condenced_result['exitcode'], - 'structuredOut': str(json.dumps(process_condenced_result['structuredOut'])) if 'structuredOut' in process_condenced_result else '', - 'status': status, - }) - - self.commandStatuses.put_command_status(handle.command, roleResult) def execute_status_command(self, command): ''' @@ -371,11 +275,4 @@ class ActionQueue(threading.Thread): """ Actions that are executed every time when command status changes """ - self.controller.heartbeat_wait_event.set() - - # Removes all commands from the queue - def reset(self): - queue = self.commandQueue - with queue.mutex: - queue.queue.clear() - + self.controller.trigger_heartbeat() http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py ---------------------------------------------------------------------- diff --git a/ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py b/ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py index bae5362..d8349ef 100644 --- a/ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py +++ b/ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py @@ -47,14 +47,13 @@ class ActualConfigHandler: filename = component + "_" + self.CONFIG_NAME self.write_file(filename, tags) - def write_client_components(self, serviceName, tags, components): + def write_client_components(self, serviceName, tags): from LiveStatus import LiveStatus for comp in LiveStatus.CLIENT_COMPONENTS: if comp['serviceName'] == serviceName: componentName = comp['componentName'] if componentName in self.configTags and \ - tags != self.configTags[componentName] and \ - (components == ["*"] or componentName in components): + tags != self.configTags[componentName]: self.write_actual_component(componentName, tags) pass @@ -91,11 +90,3 @@ class ActualConfigHandler: self.configTags[componentName] = \ self.read_file(componentName + "_" + self.CONFIG_NAME) return self.configTags[componentName] - - def update_component_tag(self, componentName, tag, version): - self.read_actual_component(componentName) - self.configTags[componentName][tag] = version - - filename = componentName + "_" + self.CONFIG_NAME - self.write_file(filename, self.configTags[componentName]) - http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/src/main/python/ambari_agent/AgentConfig_linux.py ---------------------------------------------------------------------- diff --git a/ambari-agent/src/main/python/ambari_agent/AgentConfig_linux.py b/ambari-agent/src/main/python/ambari_agent/AgentConfig_linux.py new file mode 100644 index 0000000..3ced245 --- /dev/null +++ b/ambari-agent/src/main/python/ambari_agent/AgentConfig_linux.py @@ -0,0 +1,232 @@ +#!/usr/bin/env python + +''' +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +''' + +import os + +content = """ + +[server] +hostname=localhost +url_port=8440 +secured_url_port=8441 + +[agent] +prefix=/tmp/ambari-agent +data_cleanup_interval=86400 +data_cleanup_max_age=2592000 +ping_port=8670 +cache_dir=/var/lib/ambari-agent/cache + +[services] + +[python] +custom_actions_dir = /var/lib/ambari-agent/resources/custom_actions + +[command] +maxretries=2 +sleepBetweenRetries=1 + +[security] +keysdir=/tmp/ambari-agent +server_crt=ca.crt +passphrase_env_var_name=AMBARI_PASSPHRASE + +[heartbeat] +state_interval = 6 +dirs=/etc/hadoop,/etc/hadoop/conf,/var/run/hadoop,/var/log/hadoop +rpms=glusterfs,openssl,wget,net-snmp,ntpd,ganglia,nagios,glusterfs +log_lines_count=300 + +""" + +imports = [ + "hdp/manifests/*.pp", + "hdp-hadoop/manifests/*.pp", + "hdp-hbase/manifests/*.pp", + "hdp-zookeeper/manifests/*.pp", + "hdp-oozie/manifests/*.pp", + "hdp-pig/manifests/*.pp", + "hdp-sqoop/manifests/*.pp", + "hdp-templeton/manifests/*.pp", + "hdp-hive/manifests/*.pp", + "hdp-hcat/manifests/*.pp", + "hdp-mysql/manifests/*.pp", + "hdp-monitor-webserver/manifests/*.pp", + "hdp-repos/manifests/*.pp" +] + +rolesToClass = { + 'GLUSTERFS': 'hdp-hadoop::glusterfs', + 'GLUSTERFS_CLIENT': 'hdp-hadoop::glusterfs_client', + 'GLUSTERFS_SERVICE_CHECK': 'hdp-hadoop::glusterfs_service_check', + 'NAMENODE': 'hdp-hadoop::namenode', + 'DATANODE': 'hdp-hadoop::datanode', + 'SECONDARY_NAMENODE': 'hdp-hadoop::snamenode', + 'JOBTRACKER': 'hdp-hadoop::jobtracker', + 'TASKTRACKER': 'hdp-hadoop::tasktracker', + 'RESOURCEMANAGER': 'hdp-yarn::resourcemanager', + 'NODEMANAGER': 'hdp-yarn::nodemanager', + 'HISTORYSERVER': 'hdp-yarn::historyserver', + 'YARN_CLIENT': 'hdp-yarn::yarn_client', + 'HDFS_CLIENT': 'hdp-hadoop::client', + 'MAPREDUCE_CLIENT': 'hdp-hadoop::client', + 'MAPREDUCE2_CLIENT': 'hdp-yarn::mapreducev2_client', + 'ZOOKEEPER_SERVER': 'hdp-zookeeper', + 'ZOOKEEPER_CLIENT': 'hdp-zookeeper::client', + 'HBASE_MASTER': 'hdp-hbase::master', + 'HBASE_REGIONSERVER': 'hdp-hbase::regionserver', + 'HBASE_CLIENT': 'hdp-hbase::client', + 'PIG': 'hdp-pig', + 'SQOOP': 'hdp-sqoop', + 'OOZIE_SERVER': 'hdp-oozie::server', + 'OOZIE_CLIENT': 'hdp-oozie::client', + 'HIVE_CLIENT': 'hdp-hive::client', + 'HCAT': 'hdp-hcat', + 'HIVE_SERVER': 'hdp-hive::server', + 'HIVE_METASTORE': 'hdp-hive::metastore', + 'MYSQL_SERVER': 'hdp-mysql::server', + 'WEBHCAT_SERVER': 'hdp-templeton::server', + 'DASHBOARD': 'hdp-dashboard', + 'NAGIOS_SERVER': 'hdp-nagios::server', + 'GANGLIA_SERVER': 'hdp-ganglia::server', + 'GANGLIA_MONITOR': 'hdp-ganglia::monitor', + 'HTTPD': 'hdp-monitor-webserver', + 'HUE_SERVER': 'hdp-hue::server', + 'HDFS_SERVICE_CHECK': 'hdp-hadoop::hdfs::service_check', + 'MAPREDUCE_SERVICE_CHECK': 'hdp-hadoop::mapred::service_check', + 'MAPREDUCE2_SERVICE_CHECK': 'hdp-yarn::mapred2::service_check', + 'ZOOKEEPER_SERVICE_CHECK': 'hdp-zookeeper::zookeeper::service_check', + 'ZOOKEEPER_QUORUM_SERVICE_CHECK': 'hdp-zookeeper::quorum::service_check', + 'HBASE_SERVICE_CHECK': 'hdp-hbase::hbase::service_check', + 'HIVE_SERVICE_CHECK': 'hdp-hive::hive::service_check', + 'HCAT_SERVICE_CHECK': 'hdp-hcat::hcat::service_check', + 'OOZIE_SERVICE_CHECK': 'hdp-oozie::oozie::service_check', + 'PIG_SERVICE_CHECK': 'hdp-pig::pig::service_check', + 'SQOOP_SERVICE_CHECK': 'hdp-sqoop::sqoop::service_check', + 'WEBHCAT_SERVICE_CHECK': 'hdp-templeton::templeton::service_check', + 'DASHBOARD_SERVICE_CHECK': 'hdp-dashboard::dashboard::service_check', + 'DECOMMISSION_DATANODE': 'hdp-hadoop::hdfs::decommission', + 'HUE_SERVICE_CHECK': 'hdp-hue::service_check', + 'RESOURCEMANAGER_SERVICE_CHECK': 'hdp-yarn::resourcemanager::service_check', + 'HISTORYSERVER_SERVICE_CHECK': 'hdp-yarn::historyserver::service_check', + 'TEZ_CLIENT': 'hdp-tez::tez_client', + 'YARN_SERVICE_CHECK': 'hdp-yarn::yarn::service_check', + 'FLUME_SERVER': 'hdp-flume', + 'JOURNALNODE': 'hdp-hadoop::journalnode', + 'ZKFC': 'hdp-hadoop::zkfc' +} + +serviceStates = { + 'START': 'running', + 'INSTALL': 'installed_and_configured', + 'STOP': 'stopped' +} + +servicesToPidNames = { + 'GLUSTERFS' : 'glusterd.pid$', + 'NAMENODE': 'hadoop-{USER}-namenode.pid$', + 'SECONDARY_NAMENODE': 'hadoop-{USER}-secondarynamenode.pid$', + 'DATANODE': 'hadoop-{USER}-datanode.pid$', + 'JOBTRACKER': 'hadoop-{USER}-jobtracker.pid$', + 'TASKTRACKER': 'hadoop-{USER}-tasktracker.pid$', + 'RESOURCEMANAGER': 'yarn-{USER}-resourcemanager.pid$', + 'NODEMANAGER': 'yarn-{USER}-nodemanager.pid$', + 'HISTORYSERVER': 'mapred-{USER}-historyserver.pid$', + 'JOURNALNODE': 'hadoop-{USER}-journalnode.pid$', + 'ZKFC': 'hadoop-{USER}-zkfc.pid$', + 'OOZIE_SERVER': 'oozie.pid', + 'ZOOKEEPER_SERVER': 'zookeeper_server.pid', + 'FLUME_SERVER': 'flume-node.pid', + 'TEMPLETON_SERVER': 'templeton.pid', + 'NAGIOS_SERVER': 'nagios.pid', + 'GANGLIA_SERVER': 'gmetad.pid', + 'GANGLIA_MONITOR': 'gmond.pid', + 'HBASE_MASTER': 'hbase-{USER}-master.pid', + 'HBASE_REGIONSERVER': 'hbase-{USER}-regionserver.pid', + 'HCATALOG_SERVER': 'webhcat.pid', + 'KERBEROS_SERVER': 'kadmind.pid', + 'HIVE_SERVER': 'hive-server.pid', + 'HIVE_METASTORE': 'hive.pid', + 'MYSQL_SERVER': 'mysqld.pid', + 'HUE_SERVER': '/var/run/hue/supervisor.pid', + 'WEBHCAT_SERVER': 'webhcat.pid', +} + +#Each service, which's pid depends on user should provide user mapping +servicesToLinuxUser = { + 'NAMENODE': 'hdfs_user', + 'SECONDARY_NAMENODE': 'hdfs_user', + 'DATANODE': 'hdfs_user', + 'JOURNALNODE': 'hdfs_user', + 'ZKFC': 'hdfs_user', + 'JOBTRACKER': 'mapred_user', + 'TASKTRACKER': 'mapred_user', + 'RESOURCEMANAGER': 'yarn_user', + 'NODEMANAGER': 'yarn_user', + 'HISTORYSERVER': 'mapred_user', + 'HBASE_MASTER': 'hbase_user', + 'HBASE_REGIONSERVER': 'hbase_user', +} + +pidPathesVars = [ + {'var' : 'glusterfs_pid_dir_prefix', + 'defaultValue' : '/var/run'}, + {'var' : 'hadoop_pid_dir_prefix', + 'defaultValue' : '/var/run/hadoop'}, + {'var' : 'hadoop_pid_dir_prefix', + 'defaultValue' : '/var/run/hadoop'}, + {'var' : 'ganglia_runtime_dir', + 'defaultValue' : '/var/run/ganglia/hdp'}, + {'var' : 'hbase_pid_dir', + 'defaultValue' : '/var/run/hbase'}, + {'var' : '', + 'defaultValue' : '/var/run/nagios'}, + {'var' : 'zk_pid_dir', + 'defaultValue' : '/var/run/zookeeper'}, + {'var' : 'oozie_pid_dir', + 'defaultValue' : '/var/run/oozie'}, + {'var' : 'hcat_pid_dir', + 'defaultValue' : '/var/run/webhcat'}, + {'var' : 'hive_pid_dir', + 'defaultValue' : '/var/run/hive'}, + {'var' : 'mysqld_pid_dir', + 'defaultValue' : '/var/run/mysqld'}, + {'var' : 'hcat_pid_dir', + 'defaultValue' : '/var/run/webhcat'}, + {'var' : 'yarn_pid_dir_prefix', + 'defaultValue' : '/var/run/hadoop-yarn'}, + {'var' : 'mapred_pid_dir_prefix', + 'defaultValue' : '/var/run/hadoop-mapreduce'}, +] + +if 'AMBARI_AGENT_CONF_DIR' in os.environ: + configFile = os.path.join(os.environ['AMBARI_AGENT_CONF_DIR'], "ambari-agent.ini") +else: + configFile = "/etc/ambari-agent/conf/ambari-agent.ini" + +if 'AMBARI_AGENT_LOG_DIR' in os.environ: + logfile = os.path.join(os.environ['AMBARI_AGENT_LOG_DIR'], "ambari-agent.log") +else: + logfile = "/var/log/ambari-agent/ambari-agent.log" + +if 'AMBARI_AGENT_OUT_DIR' in os.environ: + outfile = os.path.join(os.environ['AMBARI_AGENT_OUT_DIR'], "ambari-agent.out") +else: + outfile = "/var/log/ambari-agent/ambari-agent.out" http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/src/main/python/ambari_agent/AgentConfig_windows.py ---------------------------------------------------------------------- diff --git a/ambari-agent/src/main/python/ambari_agent/AgentConfig_windows.py b/ambari-agent/src/main/python/ambari_agent/AgentConfig_windows.py new file mode 100644 index 0000000..935b466 --- /dev/null +++ b/ambari-agent/src/main/python/ambari_agent/AgentConfig_windows.py @@ -0,0 +1,232 @@ +#!/usr/bin/env python + +''' +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +''' + +import os + +content = """ + +[server] +hostname=localhost +url_port=8440 +secured_url_port=8441 + +[agent] +prefix=\\tmp\\ambari-agent +data_cleanup_interval=86400 +data_cleanup_max_age=2592000 +ping_port=8670 +cache_dir=\\var\\lib\\ambari-agent\\cache + +[services] + +[python] +custom_actions_dir = \\var\\lib\\ambari-agent\\resources\\custom_actions + +[command] +maxretries=2 +sleepBetweenRetries=1 + +[security] +keysdir=\\tmp\\ambari-agent +server_crt=ca.crt +passphrase_env_var_name=AMBARI_PASSPHRASE + +[heartbeat] +state_interval = 6 +dirs=\\etc\\hadoop,\\etc\\hadoop\\conf,\\var\\run\\hadoop,\\var\\log\\hadoop +rpms=glusterfs,openssl,wget,net-snmp,ntpd,ganglia,nagios,glusterfs +log_lines_count=300 + +""" + +imports = [ + "hdp\\manifests\\*.pp", + "hdp-hadoop\\manifests\\*.pp", + "hdp-hbase\\manifests\\*.pp", + "hdp-zookeeper\\manifests\\*.pp", + "hdp-oozie\\manifests\\*.pp", + "hdp-pig\\manifests\\*.pp", + "hdp-sqoop\\manifests\\*.pp", + "hdp-templeton\\manifests\\*.pp", + "hdp-hive\\manifests\\*.pp", + "hdp-hcat\\manifests\\*.pp", + "hdp-mysql\\manifests\\*.pp", + "hdp-monitor-webserver\\manifests\\*.pp", + "hdp-repos\\manifests\\*.pp" +] + +rolesToClass = { + 'GLUSTERFS': 'hdp-hadoop::glusterfs', + 'GLUSTERFS_CLIENT': 'hdp-hadoop::glusterfs_client', + 'GLUSTERFS_SERVICE_CHECK': 'hdp-hadoop::glusterfs_service_check', + 'NAMENODE': 'hdp-hadoop::namenode', + 'DATANODE': 'hdp-hadoop::datanode', + 'SECONDARY_NAMENODE': 'hdp-hadoop::snamenode', + 'JOBTRACKER': 'hdp-hadoop::jobtracker', + 'TASKTRACKER': 'hdp-hadoop::tasktracker', + 'RESOURCEMANAGER': 'hdp-yarn::resourcemanager', + 'NODEMANAGER': 'hdp-yarn::nodemanager', + 'HISTORYSERVER': 'hdp-yarn::historyserver', + 'YARN_CLIENT': 'hdp-yarn::yarn_client', + 'HDFS_CLIENT': 'hdp-hadoop::client', + 'MAPREDUCE_CLIENT': 'hdp-hadoop::client', + 'MAPREDUCE2_CLIENT': 'hdp-yarn::mapreducev2_client', + 'ZOOKEEPER_SERVER': 'hdp-zookeeper', + 'ZOOKEEPER_CLIENT': 'hdp-zookeeper::client', + 'HBASE_MASTER': 'hdp-hbase::master', + 'HBASE_REGIONSERVER': 'hdp-hbase::regionserver', + 'HBASE_CLIENT': 'hdp-hbase::client', + 'PIG': 'hdp-pig', + 'SQOOP': 'hdp-sqoop', + 'OOZIE_SERVER': 'hdp-oozie::server', + 'OOZIE_CLIENT': 'hdp-oozie::client', + 'HIVE_CLIENT': 'hdp-hive::client', + 'HCAT': 'hdp-hcat', + 'HIVE_SERVER': 'hdp-hive::server', + 'HIVE_METASTORE': 'hdp-hive::metastore', + 'MYSQL_SERVER': 'hdp-mysql::server', + 'WEBHCAT_SERVER': 'hdp-templeton::server', + 'DASHBOARD': 'hdp-dashboard', + 'NAGIOS_SERVER': 'hdp-nagios::server', + 'GANGLIA_SERVER': 'hdp-ganglia::server', + 'GANGLIA_MONITOR': 'hdp-ganglia::monitor', + 'HTTPD': 'hdp-monitor-webserver', + 'HUE_SERVER': 'hdp-hue::server', + 'HDFS_SERVICE_CHECK': 'hdp-hadoop::hdfs::service_check', + 'MAPREDUCE_SERVICE_CHECK': 'hdp-hadoop::mapred::service_check', + 'MAPREDUCE2_SERVICE_CHECK': 'hdp-yarn::mapred2::service_check', + 'ZOOKEEPER_SERVICE_CHECK': 'hdp-zookeeper::zookeeper::service_check', + 'ZOOKEEPER_QUORUM_SERVICE_CHECK': 'hdp-zookeeper::quorum::service_check', + 'HBASE_SERVICE_CHECK': 'hdp-hbase::hbase::service_check', + 'HIVE_SERVICE_CHECK': 'hdp-hive::hive::service_check', + 'HCAT_SERVICE_CHECK': 'hdp-hcat::hcat::service_check', + 'OOZIE_SERVICE_CHECK': 'hdp-oozie::oozie::service_check', + 'PIG_SERVICE_CHECK': 'hdp-pig::pig::service_check', + 'SQOOP_SERVICE_CHECK': 'hdp-sqoop::sqoop::service_check', + 'WEBHCAT_SERVICE_CHECK': 'hdp-templeton::templeton::service_check', + 'DASHBOARD_SERVICE_CHECK': 'hdp-dashboard::dashboard::service_check', + 'DECOMMISSION_DATANODE': 'hdp-hadoop::hdfs::decommission', + 'HUE_SERVICE_CHECK': 'hdp-hue::service_check', + 'RESOURCEMANAGER_SERVICE_CHECK': 'hdp-yarn::resourcemanager::service_check', + 'HISTORYSERVER_SERVICE_CHECK': 'hdp-yarn::historyserver::service_check', + 'TEZ_CLIENT': 'hdp-tez::tez_client', + 'YARN_SERVICE_CHECK': 'hdp-yarn::yarn::service_check', + 'FLUME_SERVER': 'hdp-flume', + 'JOURNALNODE': 'hdp-hadoop::journalnode', + 'ZKFC': 'hdp-hadoop::zkfc' +} + +serviceStates = { + 'START': 'running', + 'INSTALL': 'installed_and_configured', + 'STOP': 'stopped' +} + +servicesToPidNames = { + 'GLUSTERFS' : 'glusterd.pid$', + 'NAMENODE': 'hadoop-{USER}-namenode.pid$', + 'SECONDARY_NAMENODE': 'hadoop-{USER}-secondarynamenode.pid$', + 'DATANODE': 'hadoop-{USER}-datanode.pid$', + 'JOBTRACKER': 'hadoop-{USER}-jobtracker.pid$', + 'TASKTRACKER': 'hadoop-{USER}-tasktracker.pid$', + 'RESOURCEMANAGER': 'yarn-{USER}-resourcemanager.pid$', + 'NODEMANAGER': 'yarn-{USER}-nodemanager.pid$', + 'HISTORYSERVER': 'mapred-{USER}-historyserver.pid$', + 'JOURNALNODE': 'hadoop-{USER}-journalnode.pid$', + 'ZKFC': 'hadoop-{USER}-zkfc.pid$', + 'OOZIE_SERVER': 'oozie.pid', + 'ZOOKEEPER_SERVER': 'zookeeper_server.pid', + 'FLUME_SERVER': 'flume-node.pid', + 'TEMPLETON_SERVER': 'templeton.pid', + 'NAGIOS_SERVER': 'nagios.pid', + 'GANGLIA_SERVER': 'gmetad.pid', + 'GANGLIA_MONITOR': 'gmond.pid', + 'HBASE_MASTER': 'hbase-{USER}-master.pid', + 'HBASE_REGIONSERVER': 'hbase-{USER}-regionserver.pid', + 'HCATALOG_SERVER': 'webhcat.pid', + 'KERBEROS_SERVER': 'kadmind.pid', + 'HIVE_SERVER': 'hive-server.pid', + 'HIVE_METASTORE': 'hive.pid', + 'MYSQL_SERVER': 'mysqld.pid', + 'HUE_SERVER': '\\var\\run\\hue\\supervisor.pid', + 'WEBHCAT_SERVER': 'webhcat.pid', +} + +#Each service, which's pid depends on user should provide user mapping +servicesToLinuxUser = { + 'NAMENODE': 'hdfs_user', + 'SECONDARY_NAMENODE': 'hdfs_user', + 'DATANODE': 'hdfs_user', + 'JOURNALNODE': 'hdfs_user', + 'ZKFC': 'hdfs_user', + 'JOBTRACKER': 'mapred_user', + 'TASKTRACKER': 'mapred_user', + 'RESOURCEMANAGER': 'yarn_user', + 'NODEMANAGER': 'yarn_user', + 'HISTORYSERVER': 'mapred_user', + 'HBASE_MASTER': 'hbase_user', + 'HBASE_REGIONSERVER': 'hbase_user', +} + +pidPathesVars = [ + {'var' : 'glusterfs_pid_dir_prefix', + 'defaultValue' : '\\var\\run'}, + {'var' : 'hadoop_pid_dir_prefix', + 'defaultValue' : '\\var\\run\\hadoop'}, + {'var' : 'hadoop_pid_dir_prefix', + 'defaultValue' : '\\var\\run\\hadoop'}, + {'var' : 'ganglia_runtime_dir', + 'defaultValue' : '\\var\\run\\ganglia\\hdp'}, + {'var' : 'hbase_pid_dir', + 'defaultValue' : '\\var\\run\\hbase'}, + {'var' : '', + 'defaultValue' : '\\var\\run\\nagios'}, + {'var' : 'zk_pid_dir', + 'defaultValue' : '\\var\\run\\zookeeper'}, + {'var' : 'oozie_pid_dir', + 'defaultValue' : '\\var\\run\\oozie'}, + {'var' : 'hcat_pid_dir', + 'defaultValue' : '\\var\\run\\webhcat'}, + {'var' : 'hive_pid_dir', + 'defaultValue' : '\\var\\run\\hive'}, + {'var' : 'mysqld_pid_dir', + 'defaultValue' : '\\var\\run\\mysqld'}, + {'var' : 'hcat_pid_dir', + 'defaultValue' : '\\var\\run\\webhcat'}, + {'var' : 'yarn_pid_dir_prefix', + 'defaultValue' : '\\var\\run\\hadoop-yarn'}, + {'var' : 'mapred_pid_dir_prefix', + 'defaultValue' : '\\var\\run\\hadoop-mapreduce'}, +] + +if 'AMBARI_AGENT_CONF_DIR' in os.environ: + configFile = os.path.join(os.environ['AMBARI_AGENT_CONF_DIR'], "ambari-agent.ini") +else: + configFile = "ambari-agent.ini" + +if 'AMBARI_AGENT_LOG_DIR' in os.environ: + logfile = os.path.join(os.environ['AMBARI_AGENT_LOG_DIR'], "ambari-agent.log") +else: + logfile = "\\var\\log\\ambari-agent-1.3.0-SNAPSHOT\\ambari-agent.log" + +if 'AMBARI_AGENT_OUT_DIR' in os.environ: + outfile = os.path.join(os.environ['AMBARI_AGENT_OUT_DIR'], "ambari-agent.out") +else: + outfile = "\\var\\log\\ambari-agent-1.3.0-SNAPSHOT\\ambari-agent.out" http://git-wip-us.apache.org/repos/asf/ambari/blob/7e28d1e3/ambari-agent/src/main/python/ambari_agent/AmbariAgent.py ---------------------------------------------------------------------- diff --git a/ambari-agent/src/main/python/ambari_agent/AmbariAgent.py b/ambari-agent/src/main/python/ambari_agent/AmbariAgent.py index 79433b7..e029620 100644 --- a/ambari-agent/src/main/python/ambari_agent/AmbariAgent.py +++ b/ambari-agent/src/main/python/ambari_agent/AmbariAgent.py @@ -23,8 +23,14 @@ import sys import subprocess from Controller import AGENT_AUTO_RESTART_EXIT_CODE -AGENT_SCRIPT = "/usr/lib/python2.6/site-packages/ambari_agent/main.py" -AGENT_PID_FILE = "/var/run/ambari-agent/ambari-agent.pid" +if os.environ.has_key("PYTHON_BIN"): + AGENT_SCRIPT = os.path.join(os.environ["PYTHON_BIN"],"site-packages/ambari_agent/main.py") +else: + AGENT_SCRIPT = "/usr/lib/python2.6/site-packages/ambari_agent/main.py" +if os.environ.has_key("AMBARI_PID_DIR"): + AGENT_SCRIPT = os.path.join(os.environ["AMBARI_PID_DIR"],"ambari-agent.pid") +else: + AGENT_PID_FILE = "/var/run/ambari-agent/ambari-agent.pid" # AGENT_AUTO_RESTART_EXIT_CODE = 77 is exit code which we return when restart_agent() is called status = AGENT_AUTO_RESTART_EXIT_CODE