incubator-ambari-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maha...@apache.org
Subject svn commit: r1387826 [1/3] - in /incubator/ambari/branches/AMBARI-666: ./ ambari-agent/src/main/puppet/ ambari-agent/src/main/puppet/manifestloader/ ambari-agent/src/main/puppet/modules/ ambari-agent/src/main/puppet/modules/configgenerator/ ambari-agen...
Date Thu, 20 Sep 2012 00:27:05 GMT
Author: mahadev
Date: Thu Sep 20 00:27:02 2012
New Revision: 1387826

URL: http://svn.apache.org/viewvc?rev=1387826&view=rev
Log:
AMBARI-759. Add puppet scripts to the agent for configuring/installing various services and add security aspects to api's and server/agent. (mahadev)

Added:
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/manifestloader/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/manifestloader/site.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/configfile.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/init.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/files/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/files/checkForFormat.sh
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/client.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/datanode.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/copyfromlocal.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/directory.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/service_check.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/init.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker/service_check.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/mapred/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/mapred/service_check.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/format.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/service_check.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/package.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/params.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/service.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/jobtracker-conn.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/master-conn.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/namenode-conn.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/smoketest.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/snamenode.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/tasktracker.pp
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/capacity-scheduler.xml.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/commons-logging.properties.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/core-site.xml.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/hadoop-env.sh.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/hadoop-metrics2.properties-GANGLIA.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/hadoop-metrics2.properties.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/hadoop-policy.xml.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/health_check.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/log4j.properties.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/mapred-queue-acls.xml.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/mapred-site.xml.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/slaves.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/templates/taskcontroller.cfg.erb
    incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/python/ambari_agent/security.py
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/orm/dao/RoleDAO.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/orm/dao/UserDAO.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RoleEntity.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UserEntity.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/security/
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/security/AmbariUserDetailsService.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/security/CertificateManager.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/security/ClientSecurityType.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/security/unsecured/
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/security/unsecured/rest/
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/security/unsecured/rest/CertificateDownload.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/resources/pass.txt
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/resources/users.ldif
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/resources/webapp/
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/resources/webapp/WEB-INF/
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/resources/webapp/WEB-INF/spring-security.xml
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/resources/webapp/WEB-INF/web.xml
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/resources/webapp/index.html
Modified:
    incubator/ambari/branches/AMBARI-666/AMBARI-666-CHANGES.txt
    incubator/ambari/branches/AMBARI-666/ambari-project/pom.xml
    incubator/ambari/branches/AMBARI-666/ambari-server/pom.xml
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/assemblies/server.xml
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
    incubator/ambari/branches/AMBARI-666/ambari-server/src/main/resources/META-INF/persistence.xml

Modified: incubator/ambari/branches/AMBARI-666/AMBARI-666-CHANGES.txt
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/AMBARI-666-CHANGES.txt?rev=1387826&r1=1387825&r2=1387826&view=diff
==============================================================================
--- incubator/ambari/branches/AMBARI-666/AMBARI-666-CHANGES.txt (original)
+++ incubator/ambari/branches/AMBARI-666/AMBARI-666-CHANGES.txt Thu Sep 20 00:27:02 2012
@@ -12,6 +12,10 @@ AMBARI-666 branch (unreleased changes)
 
   NEW FEATURES
 
+  AMBARI-759. Add puppet scripts to the agent for configuring/installing
+  various services and add security aspects to api's and server/agent.
+  (mahadev)
+
   AMBARI-749. Complete Java side implementation of bootstrapping agent hosts.
   (mahadev)
 

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/manifestloader/site.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/manifestloader/site.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/manifestloader/site.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/manifestloader/site.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,48 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+		
+class manifestloader () {
+    file { '/etc/puppet/agent/modules.tgz':
+      ensure => present,
+      source => "puppet:///modules/catalog/modules.tgz",  
+      mode => '0755',
+    }
+
+    exec { 'untar_modules':
+      command => "rm -rf /etc/puppet/agent/modules ; tar zxf /etc/puppet/agent/modules.tgz -C /etc/puppet/agent/ --strip-components 3",
+      path    => '/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'
+    } 
+
+    exec { 'puppet_apply':
+      command   => "sh /etc/puppet/agent/modules/puppetApply.sh",
+      timeout   => 1800,
+      path      => '/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin',
+      logoutput => "true"
+    }
+
+    File['/etc/puppet/agent/modules.tgz'] -> Exec['untar_modules'] -> Exec['puppet_apply']
+}
+
+node default {
+ stage{1 :}
+ class {'manifestloader': stage => 1}
+}
+

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/configfile.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/configfile.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/configfile.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/configfile.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,54 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+
+#
+# Generates xml configs from the given key-value hash maps
+#
+# Config file format:
+#
+# <configuration>
+#   <property>
+#     <name>name1</name><value>value1</value>
+#   </property>
+#     ..
+#   <property>
+#     <name>nameN</name><value>valueN</value>
+#   </property>
+# </configuration>
+#
+
+define configgenerator::configfile ($configname=$title, $module, $properties) {
+    $configcontent => inline_template( "
+	  <configuration>
+	  <% properties.each do |key,value| -%>
+     <property>
+	  <name><%=key %></name><value><%=value %></value>
+     </property>
+	  <% end -%>
+	  </configuration>
+	")
+ 
+	file {'config':
+       ensure  => present,
+       content => $configcontent,
+		path => "/etc/puppet/agent/modules/${module}/templates/${configname}",
+     }
+	) 

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/init.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/init.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/init.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/configgenerator/manifests/init.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,23 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+
+class configgenerator() {
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/files/checkForFormat.sh
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/files/checkForFormat.sh?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/files/checkForFormat.sh (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/files/checkForFormat.sh Thu Sep 20 00:27:02 2012
@@ -0,0 +1,62 @@
+#!/bin/sh
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+
+export hdfs_user=$1
+shift
+export conf_dir=$1
+shift
+export mark_dir=$1
+shift
+export name_dirs=$*
+
+export EXIT_CODE=0
+export command="namenode -format"
+export list_of_non_empty_dirs=""
+
+mark_file=/var/run/hadoop/hdfs/namenode-formatted
+if [[ -f ${mark_file} ]] ; then
+  rm -f ${mark_file}
+  mkdir -p ${mark_dir}
+fi
+
+if [[ ! -d $mark_dir ]] ; then
+  for dir in `echo $name_dirs | tr ',' ' '` ; do
+    echo "NameNode Dirname = $dir"
+    cmd="ls $dir | wc -l  | grep -q ^0$"
+    eval $cmd
+    if [[ $? -ne 0 ]] ; then
+      (( EXIT_CODE = $EXIT_CODE + 1 ))
+      list_of_non_empty_dirs="$list_of_non_empty_dirs $dir"
+    fi
+  done
+
+  if [[ $EXIT_CODE == 0 ]] ; then
+    su - ${hdfs_user} -c "yes Y | hadoop --config ${conf_dir} ${command}"
+  else
+    echo "ERROR: Namenode directory(s) is non empty. Will not format the namenode. List of non-empty namenode dirs ${list_of_non_empty_dirs}"
+  fi
+else
+  echo "${mark_dir} exists. Namenode DFS already formatted"
+fi
+
+exit $EXIT_CODE
+

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/client.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/client.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/client.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/client.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,56 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::client(
+  $service_state = $hdp::params::cluster_client_state
+) inherits hdp::params
+{
+  $hdp::params::service_exists['hdp-hadoop::client'] = true
+
+  Hdp-hadoop::Common<||>{service_states +> $service_state}
+
+  if ($hdp::params::use_32_bits_on_slaves == true) {
+    Hdp-hadoop::Package<||>{include_32_bit => true}
+    Hdp-hadoop::Configfile<||>{sizes +> 32}
+  } else {
+    Hdp-hadoop::Package<||>{include_64_bit => true}
+    Hdp-hadoop::Configfile<||>{sizes +> 64}
+  }
+
+  if ($service_state == 'no_op') {
+  } elsif ($service_state in ['installed_and_configured','uninstalled']) {
+    #adds package, users and directories, and common hadoop configs
+    include hdp-hadoop::initialize
+
+    if ( ($service_state == 'installed_and_configured') and
+         ($security_enabled == true) and ($kerberos_install_type == "AMBARI_SET_KERBEROS") ) {
+      $masterHost = $kerberos_adminclient_host[0]
+      hdp::download_keytab { 'hadoop_client_ambari_qa_keytab' :
+        masterhost => $masterHost,
+        keytabdst => "${$keytab_path}/ambari_qa.headless.keytab",
+        keytabfile => 'ambari_qa.headless.keytab',
+        owner => 'ambari_qa',
+        hostnameInPrincipals => 'no'
+      }
+    }
+  } else {
+    hdp_fail("TODO not implemented yet: service_state = ${service_state}")
+  }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/datanode.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/datanode.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/datanode.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/datanode.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,100 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::datanode(
+  $service_state = $hdp::params::cluster_service_state,
+  $opts = {}
+) inherits hdp-hadoop::params 
+{
+
+  $hdp::params::service_exists['hdp-hadoop::datanode'] = true
+
+  Hdp-hadoop::Common<||>{service_states +> $service_state}
+
+  if ($hdp::params::use_32_bits_on_slaves == true) {
+    Hdp-hadoop::Package<||>{include_32_bit => true}
+    Hdp-hadoop::Configfile<||>{sizes +> 32}
+  } else {
+    Hdp-hadoop::Package<||>{include_64_bit => true}
+    Hdp-hadoop::Configfile<||>{sizes +> 64}
+  }
+
+  if ($service_state == 'no_op') {
+  } elsif ($service_state in ['running','stopped','installed_and_configured','uninstalled']) { 
+    $dfs_data_dir = $hdp-hadoop::params::dfs_data_dir
+  
+    if (($hdp::params::service_exists['hdp-hadoop::namenode'] == true) or ($hdp::params::service_exists['hdp-hadoop::snamenode'] == true)){
+      $a_namenode_on_node = true
+    } else {
+      $a_namenode_on_node = false
+    }
+
+    #adds package, users and directories, and common hadoop configs
+    include hdp-hadoop::initialize
+
+    if ( ($service_state == 'installed_and_configured') and
+         ($security_enabled == true) and ($kerberos_install_type == "AMBARI_SET_KERBEROS") ) {
+      $masterHost = $kerberos_adminclient_host[0]
+      hdp::download_keytab { 'datanode_service_keytab' :
+        masterhost => $masterHost,
+        keytabdst => "${$keytab_path}/dn.service.keytab",
+        keytabfile => 'dn.service.keytab',
+        owner => $hdp-hadoop::params::hdfs_user
+      }
+    }
+
+  
+    hdp-hadoop::datanode::create_data_dirs { $dfs_data_dir: 
+      service_state => $service_state
+    }
+
+    if ($a_namenode_on_node == true){
+      $create_pid_dir = false
+      $create_log_dir = false
+    } else {
+      $create_pid_dir = true
+      $create_log_dir = true
+    }
+    
+    hdp-hadoop::service{ 'datanode':
+      ensure         => $service_state,
+      user           => $hdp-hadoop::params::hdfs_user,
+      create_pid_dir => $create_pid_dir,
+      create_log_dir => $create_log_dir
+    }
+    
+    #top level does not need anchors
+    Class['hdp-hadoop'] -> Hdp-hadoop::Service['datanode']
+    Hdp-hadoop::Datanode::Create_data_dirs<||> -> Hdp-hadoop::Service['datanode']
+  } else {
+    hdp_fail("TODO not implemented yet: service_state = ${service_state}")
+  }
+}
+
+define hdp-hadoop::datanode::create_data_dirs($service_state)
+{
+  $dirs = hdp_array_from_comma_list($name)
+  hdp::directory_recursive_create { $dirs :
+    owner => $hdp-hadoop::params::hdfs_user,
+    mode => '0750',
+    service_state => $service_state,
+    force => true
+  }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/copyfromlocal.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/copyfromlocal.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/copyfromlocal.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/copyfromlocal.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,77 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+define hdp-hadoop::hdfs::copyfromlocal(
+  $service_state,
+  $owner = unset,
+  $group = unset,
+  $recursive_chown = false,
+  $mode = undef,
+  $recursive_chmod = false,
+  $dest_dir = undef 
+) 
+{
+ 
+  if ($service_state == 'running') {
+    $copy_cmd = "fs -copyFromLocal ${name} ${dest_dir}"
+    hdp-hadoop::exec-hadoop { $copy_cmd:
+      command => $copy_cmd,
+      unless => "hadoop fs -ls ${dest_dir} >/dev/null 2>&1",
+      user => $owner
+    }
+    if ($owner == unset) {
+      $chown = ""
+    } else {
+      if ($group == unset) {
+        $chown = $owner
+      } else {
+        $chown = "${owner}:${group}"
+     } 
+    }  
+ 
+    if (chown != "") {
+      #TODO: see if there is a good 'unless test'
+      if ($recursive_chown == true) {
+        $chown_cmd = "fs -chown -R ${chown} ${dest_dir}"
+      } else {
+        $chown_cmd = "fs -chown ${chown} ${dest_dir}"
+      }
+      hdp-hadoop::exec-hadoop {$chown_cmd :
+        command => $chown_cmd,
+        user => $owner
+      }
+      Hdp-hadoop::Exec-hadoop[$copy_cmd] -> Hdp-hadoop::Exec-hadoop[$chown_cmd]
+    }
+  
+    if ($mode != undef) {
+      #TODO: see if there is a good 'unless test'
+      if ($recursive_mode == true) {
+        $chmod_cmd = "fs -chmod -R ${mode} ${dest_dir}"
+      } else {
+        $chmod_cmd = "fs -chmod ${mode} ${dest_dir}"
+      }
+      hdp-hadoop::exec-hadoop {$chmod_cmd :
+        command => $chmod_cmd,
+        user => $owner
+      }
+      Hdp-hadoop::Exec-hadoop[$copy_cmd] -> Hdp-hadoop::Exec-hadoop[$chmod_cmd]
+    }
+  }       
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/directory.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/directory.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/directory.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/directory.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,74 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+#TODO: unset should br changed to undef; just to be consistent
+define hdp-hadoop::hdfs::directory(
+  $service_state = 'running',
+  $owner = unset,
+  $group = unset,
+  $recursive_chown = false,
+  $mode = undef,
+  $recursive_chmod = false
+) 
+{
+ 
+  if ($service_state == 'running') {
+    $mkdir_cmd = "fs -mkdir ${name}"
+    hdp-hadoop::exec-hadoop { $mkdir_cmd:
+      command => $mkdir_cmd,
+      unless => "hadoop fs -ls ${name} >/dev/null 2>&1"
+    }
+    if ($owner == unset) {
+      $chown = ""
+    } else {
+      if ($group == unset) {
+        $chown = $owner
+      } else {
+        $chown = "${owner}:${group}"
+     } 
+    }  
+ 
+    if (chown != "") {
+      #TODO: see if there is a good 'unless test'
+      if ($recursive_chown == true) {
+        $chown_cmd = "fs -chown -R ${chown} ${name}"
+      } else {
+        $chown_cmd = "fs -chown ${chown} ${name}"
+      }
+      hdp-hadoop::exec-hadoop {$chown_cmd :
+        command => $chown_cmd
+      }
+      Hdp-hadoop::Exec-hadoop[$mkdir_cmd] -> Hdp-hadoop::Exec-hadoop[$chown_cmd]
+    }
+  
+    if ($mode != undef) {
+      #TODO: see if there is a good 'unless test'
+      if ($recursive_mode == true) {
+        $chmod_cmd = "fs -chmod -R ${mode} ${name}"
+      } else {
+        $chmod_cmd = "fs -chmod ${mode} ${name}"
+      }
+      hdp-hadoop::exec-hadoop {$chmod_cmd :
+        command => $chmod_cmd
+      }
+      Hdp-hadoop::Exec-hadoop[$mkdir_cmd] -> Hdp-hadoop::Exec-hadoop[$chmod_cmd]
+    }
+  }       
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/service_check.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/service_check.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/service_check.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/hdfs/service_check.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,83 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::hdfs::service_check()
+{
+  $unique = hdp_unique_id_and_date()
+  $dir = '/tmp'
+  $tmp_file = "${dir}/${unique}"
+
+  $safemode_command = "dfsadmin -safemode get | grep OFF"
+
+  $create_dir_cmd = "fs -mkdir ${dir} ; hadoop fs -chmod -R 777 ${dir}"
+  $test_dir_exists = "hadoop fs -test -e ${dir}" #TODO: may fix up fact that test needs explicit hadoop while omamnd does not
+  $cleanup_cmd = "fs -rm ${tmp_file}"
+  #cleanup put below to handle retries; if retrying there wil be a stale file that needs cleanup; exit code is fn of second command
+  $create_file_cmd = "${cleanup_cmd}; hadoop fs -put /etc/passwd ${tmp_file}" #TODO: inconsistent that second comamnd needs hadoop
+  $test_cmd = "fs -test -e ${tmp_file}"
+
+  anchor { 'hdp-hadoop::hdfs::service_check::begin':}
+
+  hdp-hadoop::exec-hadoop { 'hdfs::service_check::check_safemode':
+    command   => $safemode_command,
+    tries     => 40,
+    try_sleep => 15,
+    logoutput => true,
+    user      => $hdp::params::smokeuser,
+    require   => Anchor['hdp-hadoop::hdfs::service_check::begin']
+  }
+
+  hdp-hadoop::exec-hadoop { 'hdfs::service_check::create_dir':
+    command   => $create_dir_cmd,
+    unless    => $test_dir_exists,
+    tries     => 3,
+    try_sleep => 5,
+    user      => $hdp::params::smokeuser,
+    require   => Hdp-hadoop::Exec-hadoop['hdfs::service_check::check_safemode']
+  }
+
+  hdp-hadoop::exec-hadoop { 'hdfs::service_check::create_file':
+    command   => $create_file_cmd,
+    tries     => 3,
+    try_sleep => 5,
+    user      => $hdp::params::smokeuser,
+    require   => Hdp-hadoop::Exec-hadoop['hdfs::service_check::create_dir'],
+    notify    => Hdp-hadoop::Exec-hadoop['hdfs::service_check::test']
+  }
+
+  hdp-hadoop::exec-hadoop { 'hdfs::service_check::test':
+    command     => $test_cmd,
+    refreshonly => true,
+    user      => $hdp::params::smokeuser,
+    require     => Hdp-hadoop::Exec-hadoop['hdfs::service_check::create_file'],
+    #notify      => Hdp-hadoop::Exec-hadoop['hdfs::service_check::cleanup']  #TODO: put in after testing
+    before      => Anchor['hdp-hadoop::hdfs::service_check::end'] #TODO: remove after testing
+  }
+
+   #TODO: put in after testing
+ #  hdp-hadoop::exec-hadoop { 'hdfs::service_check::cleanup':
+ #   command     => $cleanup_cmd,
+ #   refreshonly => true,
+ #   require     => Hdp-hadoop::Exec-hadoop['hdfs::service_check::test'],
+ #   before      => Anchor['hdp-hadoop::hdfs::service_check::end']
+  #}
+  anchor{ 'hdp-hadoop::hdfs::service_check::end':}
+
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/init.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/init.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/init.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/init.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,243 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+#singleton for use with <||> form so that namenode, datanode, etc can pass state to hdp-hadoop and still use include
+define hdp-hadoop::common(
+  $service_states = []
+)
+{
+  class { 'hdp-hadoop':
+    service_states => $service_states    
+  }
+  anchor{'hdp-hadoop::common::begin':} -> Class['hdp-hadoop'] -> anchor{'hdp-hadoop::common::end':} 
+}
+
+class hdp-hadoop::initialize()
+{
+  if ($hdp::params::component_exists['hdp-hadoop'] == true) {
+  } else {
+    $hdp::params::component_exists['hdp-hadoop'] = true
+  }
+  hdp-hadoop::common { 'common':}
+  anchor{'hdp-hadoop::initialize::begin':} -> Hdp-hadoop::Common['common'] -> anchor{'hdp-hadoop::initialize::end':}
+}
+
+class hdp-hadoop(
+  $service_states  = []
+)
+{
+  include configgenerator
+
+  configgenerator::configfile('hdfs-site.xml': 
+    module => 'hdp-hadoop',
+    properties => {'dfs.name.dir' => '<%=scope.function_hdp_template_var("dfs_name_dir")%>',
+      'dfs.support.append' => '<%=scope.function_hdp_template_var("dfs_support_append")%>',
+      'dfs.webhdfs.enabled' => '<%=scope.function_hdp_template_var("dfs_webhdfs_enabled")%>',
+      'dfs.datanode.failed.volumes.tolerated' => '<%=scope.function_hdp_template_var("dfs_datanode_failed_volume_tolerated")%>',
+      'dfs.block.local-path-access.user' => '<%=scope.function_hdp_template_var("dfs_block_local_path_access_user")%>',
+      'dfs.data.dir' => '<%=scope.function_hdp_template_var("dfs_data_dir")%>'},)
+  
+  include hdp-hadoop::params
+  $hadoop_config_dir = $hdp-hadoop::params::conf_dir
+  $mapred_user = $hdp-hadoop::params::mapred_user  
+  $hdfs_user = $hdp-hadoop::params::hdfs_user  
+
+  anchor{'hdp-hadoop::begin':} 
+  anchor{'hdp-hadoop::end':} 
+
+  if ('uninstalled' in $service_states) {
+    hdp-hadoop::package { 'hadoop':
+      ensure => 'uninstalled'
+    }
+
+    hdp::directory_recursive_create { $hadoop_config_dir:
+      service_state => $service_state,
+      force => true
+    }
+
+    Anchor['hdp-hadoop::begin'] -> Hdp-hadoop::Package<||> -> Hdp::Directory_recursive_create[$hadoop_config_dir] -> Anchor['hdp-hadoop::end']
+  } else {
+    
+    hdp-hadoop::package { 'hadoop':}
+
+
+    hdp::directory_recursive_create { $hadoop_config_dir:
+      service_state => $service_state,
+      force => true
+    }
+ 
+    hdp::user{ $hdfs_user:}
+    hdp::user { $mapred_user:}
+
+    $logdirprefix = $hdp-hadoop::params::hadoop_logdirprefix
+    hdp::directory_recursive_create { $logdirprefix: 
+        owner => 'root'
+    }
+    $piddirprefix = $hdp-hadoop::params::hadoop_piddirprefix
+    hdp::directory_recursive_create { $piddirprefix: 
+        owner => 'root'
+    }
+ 
+    #taskcontroller.cfg properties conditional on security
+    if ($hdp::params::security_enabled == true) {
+      file { "${hdp::params::hadoop_bin}/task-controller":
+        owner   => 'root',
+        group   => $hdp::params::hadoop_user_group,
+        mode    => '6050',
+        require => Hdp-hadoop::Package['hadoop'],
+        before  => Anchor['hdp-hadoop::end']
+      }
+      $tc_owner = 'root'
+      $tc_mode = '0400'
+    } else {
+      $tc_owner = $hdfs_user
+      $tc_mode = undef
+    }
+    hdp-hadoop::configfile { 'taskcontroller.cfg' :
+      tag   => 'common',
+      owner => $tc_owner,
+      mode  => $tc_mode
+    }
+
+    $template_files = ['hadoop-env.sh','core-site.xml','hadoop-policy.xml','health_check','capacity-scheduler.xml','commons-logging.properties','log4j.properties','mapred-queue-acls.xml','slaves']
+    hdp-hadoop::configfile { $template_files:
+      tag   => 'common', 
+      owner => $hdfs_user
+    }
+    
+    hdp-hadoop::configfile { 'hadoop-metrics2.properties' : 
+      tag   => 'common', 
+      owner => $hdfs_user,
+    }
+
+    hdp-hadoop::configfile { 'mapred-site.xml': 
+      tag => 'common', 
+      owner => $mapred_user
+    }
+
+    Anchor['hdp-hadoop::begin'] -> Hdp-hadoop::Package<||> ->  Hdp::Directory_recursive_create[$hadoop_config_dir] ->  Hdp::User<|title == $hdfs_user or title == $mapred_user|> 
+    -> Hdp-hadoop::Configfile<|tag == 'common'|> -> Anchor['hdp-hadoop::end']
+    Anchor['hdp-hadoop::begin'] -> Hdp::Directory_recursive_create[$logdirprefix] -> Anchor['hdp-hadoop::end']
+    Anchor['hdp-hadoop::begin'] -> Hdp::Directory_recursive_create[$piddirprefix] -> Anchor['hdp-hadoop::end']
+  }
+}
+
+class hdp-hadoop::enable-ganglia()
+{
+  Hdp-hadoop::Configfile<|title  == 'hadoop-metrics2.properties'|>{template_tag => 'GANGLIA'}
+}
+
+###config file helper
+define hdp-hadoop::configfile(
+  $owner = undef,
+  $hadoop_conf_dir = $hdp-hadoop::params::conf_dir,
+  $mode = undef,
+  $namenode_host = undef,
+  $jtnode_host = undef,
+  $snamenode_host = undef,
+  $template_tag = undef,
+  $size = undef, #TODO: deprecate
+  $sizes = []
+) 
+{
+  #TODO: may need to be fixed 
+  if ($jtnode_host == undef) {
+    $calc_jtnode_host = $namenode_host
+  } else {
+    $calc_jtnode_host = $jtnode_host 
+  }
+ 
+  #only set 32 if theer is a 32 bit component and no 64 bit components
+  if (64 in $sizes) {
+    $common_size = 64
+  } elsif (32 in $sizes) {
+    $common_size = 32
+  } else {
+    $common_size = 6
+  }
+  
+  hdp::configfile { "${hadoop_conf_dir}/${name}":
+    component      => 'hadoop',
+    owner          => $owner,
+    mode           => $mode,
+    namenode_host  => $namenode_host,
+    snamenode_host => $snamenode_host,
+    jtnode_host    => $calc_jtnode_host,
+    template_tag   => $template_tag,
+    size           => $common_size
+  }
+}
+
+#####
+define hdp-hadoop::exec-hadoop(
+  $command,
+  $unless = undef,
+  $refreshonly = undef,
+  $echo_yes = false,
+  $kinit_override = false,
+  $tries = 1,
+  $timeout = 900,
+  $try_sleep = undef,
+  $user = undef,
+  $logoutput = undef
+)
+{
+  include hdp-hadoop::params
+  $security_enabled = $hdp::params::security_enabled
+  $conf_dir = $hdp-hadoop::params::conf_dir
+  $hdfs_user = $hdp-hadoop::params::hdfs_user
+
+  if ($user == undef) {
+    $run_user = $hdfs_user
+  } else {
+    $run_user = $user
+  }
+
+  if (($security_enabled == true) and ($kinit_override == false)) {
+    #TODO: may figure out so dont need to call kinit if auth in caceh already
+    if ($run_user in [$hdfs_user,'root']) {
+      $keytab = "${hdp-hadoop::params::keytab_path}/${hdfs_user}.headless.keytab"
+      $principal = $hdfs_user
+    } else {
+      $keytab = "${hdp-hadoop::params::keytab_path}/${user}.headless.keytab" 
+      $principal = $user
+    }
+    $kinit_if_needed = "/usr/kerberos/bin/kinit  -kt ${keytab} ${principal}; "
+  } else {
+    $kinit_if_needed = ""
+  }
+ 
+  if ($echo_yes == true) {
+    $cmd = "${kinit_if_needed}yes Y | hadoop --config ${conf_dir} ${command}"
+  } else {
+    $cmd = "${kinit_if_needed}hadoop --config ${conf_dir} ${command}"
+  }
+
+  hdp::exec { $cmd:
+    command     => $cmd,
+    user        => $run_user,
+    unless      => $unless,
+    refreshonly => $refreshonly,
+    tries       => $tries,
+    timeout     => $timeout,
+    try_sleep   => $try_sleep,
+    logoutput   => $logoutput
+  }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,94 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::jobtracker(
+  $service_state = $hdp::params::cluster_service_state,
+  $opts = {}
+) inherits hdp-hadoop::params
+{
+  $hdp::params::service_exists['hdp-hadoop::jobtracker'] = true
+  Hdp-hadoop::Common<||>{service_states +> $service_state}
+  Hdp-hadoop::Package<||>{include_64_bit => true}
+  Hdp-hadoop::Configfile<||>{sizes +> 64}
+
+  if ($service_state == 'no_op') {
+  } elsif ($service_state in ['running','stopped','installed_and_configured','uninstalled']) {
+    $mapred_user = $hdp-hadoop::params::mapred_user
+    $mapred_local_dir = $hdp-hadoop::params::mapred_local_dir 
+  
+    #adds package, users and directories, and common hadoop configs
+    include hdp-hadoop::initialize
+
+    if ( ($service_state == 'installed_and_configured') and
+         ($security_enabled == true) and ($kerberos_install_type == "AMBARI_SET_KERBEROS") ) {
+      $masterHost = $kerberos_adminclient_host[0]
+      hdp::download_keytab { 'jobtracker_service_keytab' :
+        masterhost => $masterHost,
+        keytabdst => "${$keytab_path}/jt.service.keytab",
+        keytabfile => 'jt.service.keytab',
+        owner => $hdp-hadoop::params::mapred_user
+      }
+    }
+     
+    hdp-hadoop::jobtracker::create_local_dirs { $mapred_local_dir: 
+      service_state => $service_state
+    }
+
+    #TODO: cleanup 
+    Hdp-Hadoop::Configfile<||>{jtnode_host => $hdp::params::host_address}
+
+    #TODO: do we keep precondition here?
+    if ($service_state == 'running' and $hdp-hadoop::params::use_preconditions == true) {
+      class { 'hdp-hadoop::hdfs::service_check':
+        before => Hdp-hadoop::Service['jobtracker'],
+        require => Class['hdp-hadoop']
+      }
+    }
+
+    hdp-hadoop::service{ 'jobtracker':
+      ensure       => $service_state,
+      user         => $mapred_user
+    }
+  
+    hdp-hadoop::service{ 'historyserver':
+      ensure         => $service_state,
+      user           => $mapred_user,
+      create_pid_dir => false,
+      create_log_dir => false
+    }
+
+    #top level does not need anchors
+    Class['hdp-hadoop'] -> Hdp-hadoop::Service['jobtracker'] -> Hdp-hadoop::Service['historyserver']
+    Hdp-hadoop::Jobtracker::Create_local_dirs<||> -> Hdp-hadoop::Service['jobtracker']
+  } else {
+    hdp_fail("TODO not implemented yet: service_state = ${service_state}")
+  }
+}
+
+define hdp-hadoop::jobtracker::create_local_dirs($service_state)
+{
+    $dirs = hdp_array_from_comma_list($name)
+    hdp::directory_recursive_create { $dirs :
+      owner => $hdp-hadoop::params::mapred_user,
+      mode => '0755',
+      service_state => $service_state,
+      force => true
+    }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker/service_check.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker/service_check.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker/service_check.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/jobtracker/service_check.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,29 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::jobtracker::service_check()
+{
+  hdp-hadoop::exec-hadoop { 'jobtracker::service_check':
+    command   => 'job -list',
+    tries     => 3,
+    try_sleep => 5,
+    user => $hdp::params::smokeuser
+  }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/mapred/service_check.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/mapred/service_check.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/mapred/service_check.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/mapred/service_check.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,75 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::mapred::service_check() 
+{
+  $smoke_test_user = $hdp::params::smokeuser
+  $jar_location = $hdp::params::hadoop_jar_location
+  $input_file = 'mapredsmokeinput'
+  $output_file = "mapredsmokeoutput"
+
+  $cleanup_cmd = "dfs -rmr ${output_file} ${input_file}"
+  #cleanup put below to handle retries; if retrying there wil be a stale file that needs cleanup; exit code is fn of second command
+  $create_file_cmd = "$cleanup_cmd ; hadoop dfs -put /etc/passwd ${input_file} " #TODO: inconsistent that second comamnd needs hadoop
+  $test_cmd = "fs -test -e ${output_file}" 
+  $run_wordcount_job = "jar ${jar_location}/hadoop-examples.jar  wordcount ${input_file} ${output_file}"
+  
+  anchor { 'hdp-hadoop::mapred::service_check::begin':}
+
+  hdp-hadoop::exec-hadoop { 'mapred::service_check::create_file':
+    command   => $create_file_cmd,
+    tries     => 1,
+    try_sleep => 5,
+    require   => Anchor['hdp-hadoop::mapred::service_check::begin'],
+  #  notify    => Hdp-hadoop::Exec-hadoop['mapred::service_check::run_wordcount'],
+    user      => $smoke_test_user
+  }
+
+  hdp-hadoop::exec-hadoop { 'mapred::service_check::run_wordcount':
+    command   => $run_wordcount_job,
+    tries     => 1,
+    try_sleep => 5,
+    require   => Hdp-hadoop::Exec-hadoop['mapred::service_check::create_file'],
+    notify    => Hdp-hadoop::Exec-hadoop['mapred::service_check::test'],
+    user      => $smoke_test_user,
+    logoutput => "true"
+  }
+
+#  exec { 'runjob':
+#    command   => "hadoop jar ${jar_location}/hadoop-examples.jar  wordcount ${input_file} ${output_file}",
+#    tries     => 1,
+#    try_sleep => 5,
+#    require   => Hdp-hadoop::Exec-hadoop['mapred::service_check::create_file'],
+#    path      => '/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin',
+#    notify    => Hdp-hadoop::Exec-hadoop['mapred::service_check::test'],
+#    logoutput => "true",
+#    user      => $smoke_test_user
+#  }
+
+  hdp-hadoop::exec-hadoop { 'mapred::service_check::test':
+    command     => $test_cmd,
+    refreshonly => true,
+    require     => Hdp-hadoop::Exec-hadoop['mapred::service_check::run_wordcount'],
+    before      => Anchor['hdp-hadoop::mapred::service_check::end'], #TODO: remove after testing
+    user        => $smoke_test_user
+  }
+  
+  anchor{ 'hdp-hadoop::mapred::service_check::end':}
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,189 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::namenode(
+  $service_state = $hdp::params::cluster_service_state,
+  $slave_hosts = [],
+  $format = true,
+  $opts = {}
+) inherits hdp-hadoop::params
+{
+  $hdp::params::service_exists['hdp-hadoop::namenode'] = true
+
+  Hdp-hadoop::Common<||>{service_states +> $service_state}
+  Hdp-hadoop::Package<||>{include_64_bit => true}
+  Hdp-hadoop::Configfile<||>{sizes +> 64}
+
+  if ($service_state == 'no_op') {
+  } elsif ($service_state in ['running','stopped','installed_and_configured','uninstalled']) { 
+    $dfs_name_dir = $hdp-hadoop::params::dfs_name_dir
+  
+    #adds package, users and directories, and common hadoop configs
+    include hdp-hadoop::initialize
+
+    if ( ($service_state == 'installed_and_configured') and 
+         ($security_enabled == true) and ($kerberos_install_type == "AMBARI_SET_KERBEROS") ) {
+      $masterHost = $kerberos_adminclient_host[0]
+      hdp::download_keytab { 'namenode_service_keytab' :
+        masterhost => $masterHost,
+        keytabdst => "${$keytab_path}/nn.service.keytab",
+        keytabfile => 'nn.service.keytab',
+        owner => $hdp-hadoop::params::hdfs_user
+      }
+      hdp::download_keytab { 'namenode_hdfs_headless_keytab' :   
+        masterhost => $masterHost,
+        keytabdst => "${$keytab_path}/hdfs.headless.keytab",
+        keytabfile => 'hdfs.headless.keytab', 
+        owner => $hdp-hadoop::params::hdfs_user, 
+        hostnameInPrincipals => 'no'
+      }
+      hdp::download_keytab { 'namenode_spnego_keytab' :   
+        masterhost => $masterHost,
+        keytabdst => "${$keytab_path}/spnego.service.keytab",
+        keytabfile => 'spnego.service.keytab', 
+        owner => $hdp-hadoop::params::hdfs_user, 
+        mode => '0440',
+        group => 'hadoop'
+      }
+    }
+ 
+    hdp-hadoop::namenode::create_name_dirs { $dfs_name_dir: 
+      service_state => $service_state
+    }
+   
+    Hdp-Hadoop::Configfile<||>{namenode_host => $hdp::params::host_address}
+    Hdp::Configfile<||>{namenode_host => $hdp::params::host_address} #for components other than hadoop (e.g., hbase) 
+  
+    if ($service_state == 'running' and $format == true) {
+      class {'hdp-hadoop::namenode::format' : }
+    }
+
+    hdp-hadoop::service{ 'namenode':
+      ensure       => $service_state,
+      user         => $hdp-hadoop::params::hdfs_user,
+      initial_wait => hdp_option_value($opts,'wait')
+    }
+
+    hdp-hadoop::namenode::create_app_directories { 'create_app_directories' :
+       service_state => $service_state
+    }
+
+    #top level does not need anchors
+    Class['hdp-hadoop'] ->  Hdp-hadoop::Service['namenode']
+    Hdp-hadoop::Namenode::Create_name_dirs<||> -> Hdp-hadoop::Service['namenode'] 
+    Hdp-hadoop::Service['namenode'] -> Hdp-hadoop::Namenode::Create_app_directories<||>
+    if ($service_state == 'running' and $format == true) {
+      Class['hdp-hadoop'] -> Class['hdp-hadoop::namenode::format'] -> Hdp-hadoop::Service['namenode']
+      Hdp-hadoop::Namenode::Create_name_dirs<||> -> Class['hdp-hadoop::namenode::format']
+    } 
+  } else {
+    hdp_fail("TODO not implemented yet: service_state = ${service_state}")
+  }
+}
+
+define hdp-hadoop::namenode::create_name_dirs($service_state)
+{
+  $dirs = hdp_array_from_comma_list($name)
+  hdp::directory_recursive_create { $dirs :
+    owner => $hdp-hadoop::params::hdfs_user,
+    mode => '0755',
+    service_state => $service_state,
+    force => true
+  }
+}
+
+define hdp-hadoop::namenode::create_app_directories($service_state)
+{
+  if ($service_state == 'running') {
+    $smoke_test_user = $hdp::params::smokeuser
+    hdp-hadoop::hdfs::directory{ "/user/${smoke_test_user}":
+      service_state => $service_state,
+      owner => $smoke_test_user,
+      mode  => '770',
+      recursive_chmod => true
+    }
+   
+    hdp-hadoop::hdfs::directory{ "/tmp" :
+      service_state => $service_state,
+      owner => $hdp-hadoop::params::hdfs_user,
+      mode => '777'
+    }
+
+    hdp-hadoop::hdfs::directory{ '/mapred' :
+      service_state => $service_state,
+      owner         => $hdp-hadoop::params::mapred_user
+    }
+    hdp-hadoop::hdfs::directory{ '/mapred/system' :
+      service_state => $service_state,
+      owner         => $hdp-hadoop::params::mapred_user
+    }
+    Hdp-hadoop::Hdfs::Directory['/mapred'] -> Hdp-hadoop::Hdfs::Directory['/mapred/system']
+
+    if ($hdp::params::hbase_master_host != "") {
+      $hdfs_root_dir = $hdp::params::hbase_hdfs_root_dir
+      hdp-hadoop::hdfs::directory { $hdfs_root_dir:
+        owner         => $hdp::params::hbase_user,
+        service_state => $service_state
+      }
+    }
+
+    if ($hdp::params::hive_server_host != "") {
+      $hive_user = $hdp::params::hive_user
+
+      hdp-hadoop::hdfs::directory{ '/apps/hive/warehouse':
+        service_state   => $service_state,
+        owner            => $hive_user,
+        mode             => '777',
+        recursive_chmod  => true
+      }
+      hdp-hadoop::hdfs::directory{ "/user/${hive_user}":
+        service_state => $service_state,
+        owner         => $hive_user
+      }
+    }
+
+    if ($hdp::params::oozie_server != "") {
+      $oozie_user = $hdp::params::oozie_user
+      hdp-hadoop::hdfs::directory{ '/user/oozie':
+        service_state => $service_state,
+        owner => $oozie_user,
+        mode  => '770',
+        recursive_chmod => true
+      }
+    }
+    
+    if ($hdp::params::templeton_server_host != "") {
+      $templeton_user = $hdp::params::templeton_user
+      hdp-hadoop::hdfs::directory{ '/user/templeton':
+        service_state => $service_state,
+        owner => $templeton_user,
+        mode  => '755',
+        recursive_chmod => true
+      }
+
+      hdp-hadoop::hdfs::directory{ '/apps/templeton':
+        service_state => $service_state,
+        owner => $templeton_user,
+        mode  => '755',
+        recursive_chmod => true
+      }
+    }
+  }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/format.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/format.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/format.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/format.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,57 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::namenode::format(
+  $force = false
+)
+{
+  $mark_dir = $hdp-hadoop::params::namenode_formatted_mark_dir
+  $dfs_name_dir = $hdp-hadoop::params::dfs_name_dir
+  $hdfs_user = $hdp::params::hdfs_user
+  $hadoop_conf_dir = $hdp-hadoop::params::conf_dir
+
+  if ($force == true) {
+      hdp-hadoop::exec-hadoop { 'namenode -format' :
+      command => 'namenode -format',
+      kinit_override => true,
+      notify  => Hdp::Exec['set namenode mark']
+    }
+  } else {
+      file { '/tmp/checkForFormat.sh':
+      ensure => present,
+      source => "puppet:///modules/hdp-hadoop/checkForFormat.sh",
+      mode => '0755'
+    }
+
+    exec { '/tmp/checkForFormat.sh':
+      command   => "sh /tmp/checkForFormat.sh ${hdfs_user} ${hadoop_conf_dir} ${mark_dir} ${dfs_name_dir} ",
+      unless   => "test -d ${mark_dir}",
+      require   => File['/tmp/checkForFormat.sh'],
+      path      => '/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin',
+      logoutput => "true",
+      notify   => Hdp::Exec['set namenode mark']
+    }
+  }
+
+  hdp::exec { 'set namenode mark' :
+    command     => "mkdir -p ${mark_dir}",
+    refreshonly => true
+  }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/service_check.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/service_check.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/service_check.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/namenode/service_check.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,28 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::namenode::service_check()
+{
+  hdp-hadoop::exec-hadoop { 'namenode::service_check':
+    command   => 'dfs -ls /',
+    tries     => 3,
+    try_sleep => 5
+  }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/package.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/package.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/package.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/package.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,44 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+#singleton, but using define so can use collections to override params
+define hdp-hadoop::package(
+  $ensure = 'present',
+  $include_32_bit = false,
+  $include_64_bit = false
+)
+{
+  #just use 32 if its specifically requested and no 64 bit requests
+  if ($include_32_bit == true) and ($include_64_bit != true) {
+    $size = 32
+  } else  {
+    $size = 64
+  }
+  $package = "hadoop ${size}"
+  $lzo_enabled = $hdp::params::lzo_enabled
+
+  hdp::package{ $package:
+    ensure       => $ensure,
+    package_type => 'hadoop',
+    size         => $size,
+    lzo_needed   => $lzo_enabled
+  }
+  anchor{ 'hdp-hadoop::package::helper::begin': } -> Hdp::Package[$package] -> anchor{ 'hdp-hadoop::package::helper::end': }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/params.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/params.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/params.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/params.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,172 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::params(
+) inherits hdp::params 
+{
+
+  ##TODO: for testing in masterless mode
+  $use_preconditions = false
+  ####  
+  $conf_dir = $hdp::params::hadoop_conf_dir 
+
+  ####### users
+
+  $mapred_user = $hdp::params::mapred_user
+  $hdfs_user = $hdp::params::hdfs_user
+  
+  ##### security related
+  $keytab_path = hdp_default("keytab_path","/etc/security/keytabs")
+ 
+  if ($hdp::params::security_enabled == true) {
+    $enable_security_authorization = true
+    $security_type = "kerberos"
+    $task_controller = "org.apache.hadoop.mapred.LinuxTaskController"
+    $dfs_datanode_address = 1019
+    $dfs_datanode_http_address = 1022
+  } else {
+    $enable_security_authorization = false
+    $security_type = "simple"
+    $task_controller = "org.apache.hadoop.mapred.DefaultTaskController"
+    $dfs_datanode_address = 50010
+    $dfs_datanode_http_address = 50075
+  }
+
+  ### hadoop-env
+  
+  $dtnode_heapsize = hdp_default("hadoop/hadoop-env/dtnode_heapsize","1024m")
+  $ttnode_heapsize = hdp_default("hadoop/hadoop-env/ttnode_heapsize","1024m")
+
+  $hadoop_heapsize = hdp_default("hadoop/hadoop-env/hadoop_heapsize","1024m")
+
+  $hadoop_logdirprefix = hdp_default("hadoop/hadoop-env/hadoop_logdirprefix","/var/log/hadoop")
+
+  $hadoop_piddirprefix = hdp_default("hadoop/hadoop-env/hadoop_piddirprefix","/var/run/hadoop")
+  $run_dir = $hadoop_piddirprefix
+
+  $namenode_formatted_mark_dir = "${run_dir}/hdfs/namenode/formatted/"
+
+  $jtnode_heapsize = hdp_default("hadoop/hadoop-env/jtnode_heapsize","1024m")
+
+  $jtnode_opt_maxnewsize = hdp_default("hadoop/hadoop-env/jtnode_opt_maxnewsize","200m")
+
+  $jtnode_opt_newsize = hdp_default("hadoop/hadoop-env/jtnode_opt_newsize","200m")
+
+  $namenode_heapsize = hdp_default("hadoop/hadoop-env/namenode_heapsize","1024m")
+
+  $namenode_opt_maxnewsize = hdp_default("hadoop/hadoop-env/namenode_opt_maxnewsize","640m")
+
+  $namenode_opt_newsize = hdp_default("hadoop/hadoop-env/namenode_opt_newsize","640m")
+  
+  ### compression related
+  if (($hdp::params::lzo_enabled == true) and ($hdp::params::snappy_enabled == true)) {
+    $mapred_compress_map_output = true
+    $compression_codecs =  "org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.SnappyCodec"
+    $mapred_map_output_compression_codec = "org.apache.hadoop.io.compress.SnappyCodec"
+  } elsif ($hdp::params::snappy_enabled == true) {
+    $mapred_compress_map_output = true
+    $compression_codecs = "org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec" 
+    $mapred_map_output_compression_codec = "org.apache.hadoop.io.compress.SnappyCodec"
+  } elsif ($hdp::params::lzo_enabled == true) {
+    $mapred_compress_map_output = true
+    $compression_codecs = "org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec"
+    $mapred_map_output_compression_codec = "com.hadoop.compression.lzo.LzoCodec"
+  } else { 
+    $mapred_compress_map_output = false
+    $compression_codecs = "org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec"
+    $mapred_map_output_compression_codec = "org.apache.hadoop.io.compress.DefaultCodec"
+  }
+
+  ### core-site
+  $fs_checkpoint_dir = hdp_default("hadoop/core-site/fs_checkpoint_dir","/tmp/dfs/namesecondary")
+
+  $proxyuser_group = hdp_default("hadoop/core-site/proxyuser_group","users")
+
+  ### hdfs-site
+  $datanode_du_reserved = hdp_default("hadoop/hdfs-site/datanode_du_reserved",1073741824)
+
+  $dfs_block_local_path_access_user = hdp_default("hadoop/hdfs-site/dfs_block_local_path_access_user","hbase")
+
+  $dfs_data_dir = $hdp::params::dfs_data_dir
+
+  $dfs_datanode_data_dir_perm = hdp_default("hadoop/hdfs-site/dfs_datanode_data_dir_perm",750)
+
+  $dfs_datanode_failed_volume_tolerated = hdp_default("hadoop/hdfs-site/dfs_datanode_failed_volume_tolerated",0)
+
+  $dfs_exclude = hdp_default("hadoop/hdfs-site/dfs_exclude","dfs.exclude")
+
+  $dfs_include = hdp_default("hadoop/hdfs-site/dfs_include","dfs.include")
+  
+  $dfs_name_dir = hdp_default("hadoop/hdfs-site/dfs_name_dir","/tmp/hadoop-hdfs/dfs/name")
+  
+  $dfs_replication = hdp_default("hadoop/hdfs-site/dfs_replication",3)
+
+  $dfs_support_append = hdp_default("hadoop/hdfs-site/dfs_support_append",true)
+
+  $dfs_webhdfs_enabled = hdp_default("hadoop/hdfs-site/dfs_webhdfs_enabled","false")
+
+
+ ######### mapred #######
+   ### mapred-site
+
+  $mapred_system_dir = '/mapred/system'
+
+  $io_sort_mb = hdp_default("hadoop/mapred-site/io_sort_mb","200")
+
+  $io_sort_spill_percent = hdp_default("hadoop/mapred-site/io_sort_spill_percent","0.9")
+
+  $mapred_child_java_opts_sz = hdp_default("hadoop/mapred-site/mapred_child_java_opts_sz","-Xmx768m")
+
+  $mapred_cluster_map_mem_mb = hdp_default("hadoop/mapred-site/mapred_cluster_map_mem_mb","-1")
+
+  $mapred_cluster_max_map_mem_mb = hdp_default("hadoop/mapred-site/mapred_cluster_max_map_mem_mb","-1")
+
+  $mapred_cluster_max_red_mem_mb = hdp_default("hadoop/mapred-site/mapred_cluster_max_red_mem_mb","-1")
+
+  $mapred_cluster_red_mem_mb = hdp_default("hadoop/mapred-site/mapred_cluster_red_mem_mb","-1")
+
+  $mapred_hosts_exclude = hdp_default("hadoop/mapred-site/mapred_hosts_exclude","mapred.exclude")
+
+  $mapred_hosts_include = hdp_default("hadoop/mapred-site/mapred_hosts_include","mapred.include")
+
+  $mapred_job_map_mem_mb = hdp_default("hadoop/mapred-site/mapred_job_map_mem_mb","-1")
+
+  $mapred_job_red_mem_mb = hdp_default("hadoop/mapred-site/mapred_job_red_mem_mb","-1")
+
+  $mapred_jobstatus_dir = hdp_default("hadoop/mapred-site/mapred_jobstatus_dir","file:////mapred/jobstatus")
+
+  $mapred_local_dir = hdp_default("hadoop/mapred-site/mapred_local_dir","/tmp/hadoop-mapred/mapred/local")
+   
+  $mapred_map_tasks_max = hdp_default("hadoop/mapred-site/mapred_map_tasks_max",4)
+
+  $mapred_red_tasks_max = hdp_default("hadoop/mapred-site/mapred_red_tasks_max",4)
+
+  $mapreduce_userlog_retainhours = hdp_default("hadoop/mapred-site/mapreduce_userlog_retainhours",24)
+
+  $maxtasks_per_job = hdp_default("hadoop/mapred-site/maxtasks_per_job","-1")
+
+  $scheduler_name = hdp_default("hadoop/mapred-site/scheduler_name","org.apache.hadoop.mapred.CapacityTaskScheduler")
+
+  #### health_check
+
+  $security_enabled = $hdp::params::security_enabled
+
+  $task_bin_exe = hdp_default("hadoop/health_check/task_bin_exe")
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/service.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/service.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/service.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/service.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,118 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+define hdp-hadoop::service(
+  $ensure = 'running',
+  $user,
+  $initial_wait = undef,
+  $create_pid_dir = true,
+  $create_log_dir = true
+)
+{
+
+  $security_enabled = $hdp::params::security_enabled
+
+  #NOTE does not work if namenode and datanode are on same host 
+  $pid_dir = "${hdp-hadoop::params::hadoop_piddirprefix}/${user}"
+  
+  if (($security_enabled == true) and ($name == 'datanode')) {
+    $run_as_root = true
+  } else {       
+    $run_as_root = false
+  }
+
+  if (($security_enabled == true) and ($name == 'datanode')) {
+    $hdfs_user = $hdp::params::hdfs_user
+    $pid_file = "${hdp-hadoop::params::hadoop_piddirprefix}/${hdfs_user}/hadoop-${hdfs_user}-${name}.pid"
+  } else {
+    $pid_file = "${pid_dir}/hadoop-${user}-${name}.pid"
+  } 
+
+  $log_dir = "${hdp-hadoop::params::hadoop_logdirprefix}/${user}"
+  $hadoop_daemon = "${hdp::params::hadoop_bin}/hadoop-daemon.sh"
+   
+  $cmd = "${hadoop_daemon} --config ${hdp-hadoop::params::conf_dir}"
+  if ($ensure == 'running') {
+    if ($run_as_root == true) {
+      $daemon_cmd = "${cmd} start ${name}"
+    } else {
+      $daemon_cmd = "su - ${user} -c  '${cmd} start ${name}'"
+    }
+    $service_is_up = "ls ${pid_file} >/dev/null 2>&1 && ps `cat ${pid_file}` >/dev/null 2>&1"
+  } elsif ($ensure == 'stopped') {
+    if ($run_as_root == true) {
+      $daemon_cmd = "${cmd} stop ${name}"
+    } else {
+      $daemon_cmd = "su - ${user} -c  '${cmd} stop ${name}'"
+    }
+    $service_is_up = undef
+  } else {
+    $daemon_cmd = undef
+  }
+ 
+  if ($create_pid_dir == true) {
+    hdp::directory_recursive_create { $pid_dir: 
+      owner       => $user,
+      context_tag => 'hadoop_service',
+      service_state => $service_state,
+      force => true
+    }
+  }
+  
+  if ($create_log_dir == true) {
+    hdp::directory_recursive_create { $log_dir: 
+      owner       => $user,
+      context_tag => 'hadoop_service',
+      service_state => $service_state,
+      force => true
+    }
+  }
+  if ($daemon_cmd != undef) {  
+    hdp::exec { $daemon_cmd:
+      command      => $daemon_cmd,
+      unless       => $service_is_up,
+      initial_wait => $initial_wait
+    }
+  }
+
+  anchor{"hdp-hadoop::service::${name}::begin":}
+  anchor{"hdp-hadoop::service::${name}::end":}
+  if ($daemon_cmd != undef) {
+    Anchor["hdp-hadoop::service::${name}::begin"] -> Hdp::Exec[$daemon_cmd] -> Anchor["hdp-hadoop::service::${name}::end"]
+
+    if ($create_pid_dir == true) {
+      Anchor["hdp-hadoop::service::${name}::begin"] -> Hdp::Directory_recursive_create[$pid_dir] -> Hdp::Exec[$daemon_cmd] 
+    }
+     if ($create_log_dir == true) {
+      Anchor["hdp-hadoop::service::${name}::begin"] -> Hdp::Directory_recursive_create[$log_dir] -> Hdp::Exec[$daemon_cmd] 
+    }
+  }
+  if ($ensure == 'running') {
+    #TODO: look at Puppet resource retry and retry_sleep
+    #TODO: can make sleep contingent on $name
+    $sleep = 5
+    $post_check = "sleep ${sleep}; ${service_is_up}"
+    hdp::exec { $post_check:
+      command => $post_check,
+      unless  => $service_is_up
+    }
+    Hdp::Exec[$daemon_cmd] -> Hdp::Exec[$post_check] -> Anchor["hdp-hadoop::service::${name}::end"]
+  }  
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/jobtracker-conn.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/jobtracker-conn.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/jobtracker-conn.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/jobtracker-conn.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,24 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::slave::jobtracker-conn($jobtracker_host)
+{
+  Hdp-Hadoop::Configfile<||>{jtnode_host => $jobtracker_host}
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/master-conn.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/master-conn.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/master-conn.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/master-conn.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,27 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::slave::master-conn($master_host)
+{
+  Hdp-Hadoop::Configfile<||>{
+    namenode_host => $master_host,
+    jtnode_host   => $master_host
+  }
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/namenode-conn.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/namenode-conn.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/namenode-conn.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/slave/namenode-conn.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,27 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+#TODO: this might be replaced by just using hdp::namenode-conn
+class hdp-hadoop::slave::namenode-conn($namenode_host)
+{
+  #TODO: check if can get rido of both
+  Hdp-Hadoop::Configfile<||>{namenode_host => $namenode_host}
+  Hdp::Configfile<||>{namenode_host => $namenode_host} #for components other than hadoop (e.g., hbase) 
+}

Added: incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/smoketest.pp
URL: http://svn.apache.org/viewvc/incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/smoketest.pp?rev=1387826&view=auto
==============================================================================
--- incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/smoketest.pp (added)
+++ incubator/ambari/branches/AMBARI-666/ambari-agent/src/main/puppet/modules/hdp-hadoop/manifests/smoketest.pp Thu Sep 20 00:27:02 2012
@@ -0,0 +1,46 @@
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#
+class hdp-hadoop::smoketest(
+  $opts={}
+)
+{
+  #TODO: put in wait
+  #TODO: look for better way to compute outname
+  $date_format = '"%M%d%y"'
+  $outname = inline_template("<%=  `date +${date_format}`.chomp %>")
+
+  #TODO: hardwired to run on namenode and to use user hdfs
+
+  $put = "dfs -put /etc/passwd passwd-${outname}"
+  $exec = "jar /usr/share/hadoop/hadoop-examples-*.jar wordcount passwd-${outname} ${outname}.out"
+  $result = "fs -test -e ${outname}.out /dev/null 2>&1"
+  anchor{ "hdp-hadoop::smoketest::begin" :} ->
+  hdp-hadoop::exec-hadoop{ $put:
+    command => $put
+  } ->
+  hdp-hadoop::exec-hadoop{ $exec:
+    command =>  $exec
+  } ->
+  hdp-hadoop::exec-hadoop{ $result:
+    command =>  $result
+  } ->
+  anchor{ "hdp-hadoop::smoketest::end" :}
+}



Mime
View raw message