ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sumit Mohanty (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-12401) Oozie Server start failed and RegionServers down after "Node Labels" enabled
Date Mon, 13 Jul 2015 21:32:04 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-12401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sumit Mohanty updated AMBARI-12401:
-----------------------------------
    Description: 
*scenario*
# install ambari
# deploy cluster
# navigate YARN configs tab
# set "Node Labels" property to "Enabled"
# restart YARN and Oozie
*expected* restart success
*actual* *Oozie* restart failed and *RegionServers* randomly shut down
{code}stderr:   /var/lib/ambari-agent/data/errors-227.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 181, in <module>
    OozieServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 459, in restart
    self.start(env)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 64, in start
    oozie_service(action='start', rolling_restart=rolling_restart)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py", line 102, in oozie_service
    list_status = util.run_command(params.hdfs_share_dir, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 201, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ip-x-x-x-x.ec2.internal:50070/webhdfs/v1/user/oozie/share?op=GETFILESTATUS&user.name=oozie'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "AccessControlException", 
    "javaClassName": "org.apache.hadoop.security.AccessControlException", 
    "message": "Permission denied: user=oozie, access=EXECUTE, inode=\"/user/oozie/share\":yarn:hadoop:drwx------"
  }
}
stdout:   /var/lib/ambari-agent/data/output-227.txt

2015-07-13 13:01:16,044 - Group['hadoop'] {'ignore_failures': False}
2015-07-13 13:01:16,060 - Group['users'] {'ignore_failures': False}
2015-07-13 13:01:16,060 - Group['knox'] {'ignore_failures': False}
2015-07-13 13:01:16,061 - Group['spark'] {'ignore_failures': False}
2015-07-13 13:01:16,061 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
2015-07-13 13:01:16,063 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,064 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
2015-07-13 13:01:16,065 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,066 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,068 - User['knox'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,069 - User['storm'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,070 - User['spark'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,071 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,073 - User['accumulo'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,074 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,075 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
2015-07-13 13:01:16,076 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,077 - User['falcon'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
2015-07-13 13:01:16,078 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,079 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,080 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,082 - User['ams'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,083 - User['atlas'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,085 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2015-07-13 13:01:16,088 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2015-07-13 13:01:16,153 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2015-07-13 13:01:16,154 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2015-07-13 13:01:16,156 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2015-07-13 13:01:16,158 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2015-07-13 13:01:16,219 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2015-07-13 13:01:16,220 - Group['hdfs'] {'ignore_failures': False}
2015-07-13 13:01:16,221 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']}
2015-07-13 13:01:16,222 - Directory['/etc/hadoop'] {'mode': 0755}
2015-07-13 13:01:16,252 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2015-07-13 13:01:16,281 - Execute['('setenforce', '0')'] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2015-07-13 13:01:16,415 - Directory['/grid/0/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2015-07-13 13:01:16,417 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2015-07-13 13:01:16,417 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2015-07-13 13:01:16,423 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2015-07-13 13:01:16,425 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2015-07-13 13:01:16,426 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2015-07-13 13:01:16,435 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2015-07-13 13:01:16,436 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2015-07-13 13:01:16,437 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2015-07-13 13:01:16,445 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'group': 'hadoop'}
2015-07-13 13:01:16,445 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'mode': 0755}
2015-07-13 13:01:16,780 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh'] {'environment': {'OOZIE_CONFIG': '/usr/hdp/current/oozie-server/conf'}, 'only_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'user': 'oozie'}
2015-07-13 13:01:16,903 - Skipping Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh'] due to only_if
2015-07-13 13:01:16,903 - File['/var/run/oozie/oozie.pid'] {'action': ['delete']}
2015-07-13 13:01:16,910 - HdfsResource['/user/oozie'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://ip-x:8020', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'oozie', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'mode': 0775}
2015-07-13 13:01:16,913 - checked_call['curl -sS -L -w '%{http_code}' -X GET 'http://ip-x:50070/webhdfs/v1/user/oozie?op=GETFILESTATUS&user.name=hdfs''] {'logoutput': None, 'user': 'hdfs', 'stderr': -1, 'quiet': False}
2015-07-13 13:01:17,046 - checked_call returned (0, '{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17109,"group":"hdfs","length":0,"modificationTime":1436785928420,"owner":"oozie","pathSuffix":"","permission":"775","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', '')
2015-07-13 13:01:17,048 - HdfsResource['None'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://-x:8020', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf'}
2015-07-13 13:01:17,049 - Directory['/usr/hdp/current/oozie-server/conf'] {'owner': 'oozie', 'group': 'hadoop', 'recursive': True}
2015-07-13 13:01:17,050 - XmlConfig['oozie-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf', 'mode': 0664, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
2015-07-13 13:01:17,086 - Generating config: /usr/hdp/current/oozie-server/conf/oozie-site.xml
2015-07-13 13:01:17,086 - File['/usr/hdp/current/oozie-server/conf/oozie-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0664, 'encoding': 'UTF-8'}
2015-07-13 13:01:17,112 - Writing File['/usr/hdp/current/oozie-server/conf/oozie-site.xml'] because contents don't match
2015-07-13 13:01:17,122 - File['/usr/hdp/current/oozie-server/conf/oozie-env.sh'] {'content': InlineTemplate(...), 'owner': 'oozie'}
2015-07-13 13:01:17,123 - Writing File['/usr/hdp/current/oozie-server/conf/oozie-env.sh'] because contents don't match
2015-07-13 13:01:17,124 - File['/usr/hdp/current/oozie-server/conf/oozie-log4j.properties'] {'content': '...', 'owner': 'oozie', 'group': 'hadoop', 'mode': 0644}
2015-07-13 13:01:17,132 - File['/usr/hdp/current/oozie-server/conf/adminusers.txt'] {'content': Template('adminusers.txt.j2'), 'owner': 'oozie', 'group': 'hadoop', 'mode': 0644}
2015-07-13 13:01:17,134 - File['/usr/lib/ambari-agent/DBConnectionVerification.jar'] {'content': DownloadSource('http://ip--x:8080/resources/DBConnectionVerification.jar')}
2015-07-13 13:01:17,134 - Not downloading the file from http://ip-x:8080/resources/DBConnectionVerification.jar, because /var/lib/ambari-agent/data/tmp/DBConnectionVerification.jar already exists
2015-07-13 13:01:17,135 - File['/usr/hdp/current/oozie-server/conf/hadoop-config.xml'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:17,136 - File['/usr/hdp/current/oozie-server/conf/oozie-default.xml'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:17,136 - Directory['/usr/hdp/current/oozie-server/conf/action-conf'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:17,137 - File['/usr/hdp/current/oozie-server/conf/action-conf/hive.xml'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:17,138 - File['/var/run/oozie/oozie.pid'] {'action': ['delete'], 'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
2015-07-13 13:01:17,244 - Directory['/usr/hdp/current/oozie-server//var/tmp/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,246 - Directory['/var/run/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,247 - Directory['/grid/0/log/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,247 - Directory['/var/tmp/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,248 - Directory['/hadoop/oozie/data'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,249 - Directory['/usr/hdp/current/oozie-server'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,249 - Directory['/usr/hdp/current/oozie-server/oozie-server/webapps'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,250 - Directory['/usr/hdp/current/oozie-server/oozie-server/conf'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,251 - Directory['/usr/hdp/current/oozie-server/oozie-server'] {'owner': 'oozie', 'recursive': True, 'group': 'hadoop', 'mode': 0755, 'cd_access': 'a'}
2015-07-13 13:01:17,251 - Directory['/usr/hdp/current/oozie-server/libext'] {'recursive': True}
2015-07-13 13:01:17,252 - Execute['('tar', '-xvf', '/usr/hdp/current/oozie-server/oozie-sharelib.tar.gz', '-C', '/usr/hdp/current/oozie-server')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1' || test -f /usr/hdp/current/oozie-server/.hashcode && test -d /usr/hdp/current/oozie-server/share && [[ `cat /usr/hdp/current/oozie-server/.hashcode` == '046a880c90fcbbfea52bec80cb88dd8f' ]]", 'sudo': True}
2015-07-13 13:01:17,423 - Skipping Execute['('tar', '-xvf', '/usr/hdp/current/oozie-server/oozie-sharelib.tar.gz', '-C', '/usr/hdp/current/oozie-server')'] due to not_if
2015-07-13 13:01:17,425 - Execute['('cp', '/usr/share/HDP-oozie/ext-2.2.zip', '/usr/hdp/current/oozie-server/libext')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'sudo': True}
2015-07-13 13:01:17,631 - Execute['('chown', 'oozie:hadoop', '/usr/hdp/current/oozie-server/libext/ext-2.2.zip')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'sudo': True}
2015-07-13 13:01:17,814 - Execute['('chown', '-RL', 'oozie:hadoop', '/usr/hdp/current/oozie-server/oozie-server/conf')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'sudo': True}
2015-07-13 13:01:17,987 - File['/var/lib/ambari-agent/data/tmp/mysql-connector-java.jar'] {'content': DownloadSource('http://ip-x:8080/resources//mysql-jdbc-driver.jar')}
2015-07-13 13:01:17,989 - Not downloading the file from http://ip-x:8080/resources//mysql-jdbc-driver.jar, because /var/lib/ambari-agent/data/tmp/mysql-jdbc-driver.jar already exists
2015-07-13 13:01:17,993 - Execute['('cp', '--remove-destination', '/var/lib/ambari-agent/data/tmp/mysql-connector-java.jar', '/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar')'] {'path': ['/bin', '/usr/bin/'], 'sudo': True}
2015-07-13 13:01:18,099 - File['/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:18,100 - Changing owner for /usr/hdp/current/oozie-server/libext/mysql-connector-java.jar from 0 to oozie
2015-07-13 13:01:18,100 - Changing group for /usr/hdp/current/oozie-server/libext/mysql-connector-java.jar from 0 to hadoop
2015-07-13 13:01:18,101 - Execute['ambari-sudo.sh cp /usr/hdp/current/falcon-client/oozie/ext/falcon-oozie-el-extension-*.jar /usr/hdp/current/oozie-server/libext'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
2015-07-13 13:01:18,350 - Execute['ambari-sudo.sh chown oozie:hadoop /usr/hdp/current/oozie-server/libext/falcon-oozie-el-extension-*.jar'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
2015-07-13 13:01:18,565 - Package['hadoop-lzo'] {}
2015-07-13 13:01:18,763 - Skipping installation of existing package hadoop-lzo
2015-07-13 13:01:18,764 - Package['lzo'] {}
2015-07-13 13:01:18,885 - Skipping installation of existing package lzo
2015-07-13 13:01:18,885 - Package['hadoop-lzo-native'] {}
2015-07-13 13:01:19,008 - Skipping installation of existing package hadoop-lzo-native
2015-07-13 13:01:19,008 - Package['hadooplzo_2_3_*'] {}
2015-07-13 13:01:19,154 - Skipping installation of existing package hadooplzo_2_3_*
2015-07-13 13:01:19,154 - Execute['ambari-sudo.sh cp /usr/hdp/current/hadoop-client/lib/hadoop-lzo*.jar /usr/hdp/current/oozie-server'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
2015-07-13 13:01:19,325 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] {'not_if': '...', 'user': 'oozie'}
2015-07-13 13:01:19,430 - Skipping Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] due to not_if
2015-07-13 13:01:19,431 - File['/usr/hdp/current/oozie-server/.hashcode'] {'content': '046a880c90fcbbfea52bec80cb88dd8f', 'mode': 0644}
2015-07-13 13:01:19,432 - File['/usr/hdp/current/oozie-server/.prepare_war_cmd'] {'content': 'cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war ', 'mode': 0644}
2015-07-13 13:01:19,432 - Directory['/usr/hdp/current/oozie-server/conf/action-conf/hive'] {'owner': 'oozie', 'group': 'hadoop', 'recursive': True}
2015-07-13 13:01:19,433 - XmlConfig['hive-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf/action-conf/hive', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
2015-07-13 13:01:19,447 - Generating config: /usr/hdp/current/oozie-server/conf/action-conf/hive/hive-site.xml
2015-07-13 13:01:19,447 - File['/usr/hdp/current/oozie-server/conf/action-conf/hive/hive-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2015-07-13 13:01:19,604 - Writing File['/usr/hdp/current/oozie-server/conf/action-conf/hive/hive-site.xml'] because contents don't match
2015-07-13 13:01:19,605 - XmlConfig['tez-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf/action-conf/hive', 'mode': 0664, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
2015-07-13 13:01:19,624 - Generating config: /usr/hdp/current/oozie-server/conf/action-conf/hive/tez-site.xml
2015-07-13 13:01:19,625 - File['/usr/hdp/current/oozie-server/conf/action-conf/hive/tez-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0664, 'encoding': 'UTF-8'}
2015-07-13 13:01:19,673 - Writing File['/usr/hdp/current/oozie-server/conf/action-conf/hive/tez-site.xml'] because contents don't match
2015-07-13 13:01:19,674 - Execute['('chown', '-R', 'oozie:hadoop', '/usr/hdp/current/oozie-server/oozie-server')'] {'sudo': True}
2015-07-13 13:01:19,798 - Execute['/usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://host/ooziedb' oozieuser [PROTECTED] com.mysql.jdbc.Driver'] {'tries': 5, 'try_sleep': 10}
2015-07-13 13:01:20,601 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/ooziedb.sh create -sqlfile oozie.sql -run'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'ignore_failures': True, 'user': 'oozie'}
2015-07-13 13:01:24,003 - checked_call['curl -sS -L -w '%{http_code}' -X GET 'http://ip-x:50070/webhdfs/v1/user/oozie/share?op=GETFILESTATUS&user.name=oozie''] {'logoutput': None, 'user': 'oozie', 'stderr': -1, 'quiet': False}
2015-07-13 13:01:24,122 - checked_call returned (0, '{"RemoteException":{"exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException","message":"Permission denied: user=oozie, access=EXECUTE, inode=\\"/user/oozie/share\\":yarn:hadoop:drwx------"}}403', ''){code}

  was:
*scenario*
# install ambari
# deploy cluster
# navigate YARN configs tab
# set "Node Labels" property to "Enabled"
# restart YARN and Oozie
*expected* restart success
*actual* *Oozie* restart failed and *RegionServers* randomly shut down
{code}stderr:   /var/lib/ambari-agent/data/errors-227.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 181, in <module>
    OozieServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 459, in restart
    self.start(env)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 64, in start
    oozie_service(action='start', rolling_restart=rolling_restart)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py", line 102, in oozie_service
    list_status = util.run_command(params.hdfs_share_dir, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 201, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ip-x-x-x-x.ec2.internal:50070/webhdfs/v1/user/oozie/share?op=GETFILESTATUS&user.name=oozie'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "AccessControlException", 
    "javaClassName": "org.apache.hadoop.security.AccessControlException", 
    "message": "Permission denied: user=oozie, access=EXECUTE, inode=\"/user/oozie/share\":yarn:hadoop:drwx------"
  }
}
stdout:   /var/lib/ambari-agent/data/output-227.txt

2015-07-13 13:01:16,044 - Group['hadoop'] {'ignore_failures': False}
2015-07-13 13:01:16,060 - Group['users'] {'ignore_failures': False}
2015-07-13 13:01:16,060 - Group['knox'] {'ignore_failures': False}
2015-07-13 13:01:16,061 - Group['spark'] {'ignore_failures': False}
2015-07-13 13:01:16,061 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
2015-07-13 13:01:16,063 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,064 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
2015-07-13 13:01:16,065 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,066 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,068 - User['knox'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,069 - User['storm'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,070 - User['spark'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,071 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,073 - User['accumulo'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,074 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,075 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
2015-07-13 13:01:16,076 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,077 - User['falcon'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
2015-07-13 13:01:16,078 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,079 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,080 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,082 - User['ams'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,083 - User['atlas'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2015-07-13 13:01:16,085 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2015-07-13 13:01:16,088 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2015-07-13 13:01:16,153 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2015-07-13 13:01:16,154 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2015-07-13 13:01:16,156 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2015-07-13 13:01:16,158 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2015-07-13 13:01:16,219 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2015-07-13 13:01:16,220 - Group['hdfs'] {'ignore_failures': False}
2015-07-13 13:01:16,221 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']}
2015-07-13 13:01:16,222 - Directory['/etc/hadoop'] {'mode': 0755}
2015-07-13 13:01:16,252 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2015-07-13 13:01:16,281 - Execute['('setenforce', '0')'] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2015-07-13 13:01:16,415 - Directory['/grid/0/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2015-07-13 13:01:16,417 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2015-07-13 13:01:16,417 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2015-07-13 13:01:16,423 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2015-07-13 13:01:16,425 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2015-07-13 13:01:16,426 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2015-07-13 13:01:16,435 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2015-07-13 13:01:16,436 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2015-07-13 13:01:16,437 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2015-07-13 13:01:16,445 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'group': 'hadoop'}
2015-07-13 13:01:16,445 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'mode': 0755}
2015-07-13 13:01:16,780 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh'] {'environment': {'OOZIE_CONFIG': '/usr/hdp/current/oozie-server/conf'}, 'only_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'user': 'oozie'}
2015-07-13 13:01:16,903 - Skipping Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh'] due to only_if
2015-07-13 13:01:16,903 - File['/var/run/oozie/oozie.pid'] {'action': ['delete']}
2015-07-13 13:01:16,910 - HdfsResource['/user/oozie'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://ip-172-31-41-161.ec2.internal:8020', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'oozie', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'mode': 0775}
2015-07-13 13:01:16,913 - checked_call['curl -sS -L -w '%{http_code}' -X GET 'http://ip-172-31-41-161.ec2.internal:50070/webhdfs/v1/user/oozie?op=GETFILESTATUS&user.name=hdfs''] {'logoutput': None, 'user': 'hdfs', 'stderr': -1, 'quiet': False}
2015-07-13 13:01:17,046 - checked_call returned (0, '{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17109,"group":"hdfs","length":0,"modificationTime":1436785928420,"owner":"oozie","pathSuffix":"","permission":"775","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', '')
2015-07-13 13:01:17,048 - HdfsResource['None'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://ip-172-31-41-161.ec2.internal:8020', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf'}
2015-07-13 13:01:17,049 - Directory['/usr/hdp/current/oozie-server/conf'] {'owner': 'oozie', 'group': 'hadoop', 'recursive': True}
2015-07-13 13:01:17,050 - XmlConfig['oozie-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf', 'mode': 0664, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
2015-07-13 13:01:17,086 - Generating config: /usr/hdp/current/oozie-server/conf/oozie-site.xml
2015-07-13 13:01:17,086 - File['/usr/hdp/current/oozie-server/conf/oozie-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0664, 'encoding': 'UTF-8'}
2015-07-13 13:01:17,112 - Writing File['/usr/hdp/current/oozie-server/conf/oozie-site.xml'] because contents don't match
2015-07-13 13:01:17,122 - File['/usr/hdp/current/oozie-server/conf/oozie-env.sh'] {'content': InlineTemplate(...), 'owner': 'oozie'}
2015-07-13 13:01:17,123 - Writing File['/usr/hdp/current/oozie-server/conf/oozie-env.sh'] because contents don't match
2015-07-13 13:01:17,124 - File['/usr/hdp/current/oozie-server/conf/oozie-log4j.properties'] {'content': '...', 'owner': 'oozie', 'group': 'hadoop', 'mode': 0644}
2015-07-13 13:01:17,132 - File['/usr/hdp/current/oozie-server/conf/adminusers.txt'] {'content': Template('adminusers.txt.j2'), 'owner': 'oozie', 'group': 'hadoop', 'mode': 0644}
2015-07-13 13:01:17,134 - File['/usr/lib/ambari-agent/DBConnectionVerification.jar'] {'content': DownloadSource('http://ip-172-31-41-161.ec2.internal:8080/resources/DBConnectionVerification.jar')}
2015-07-13 13:01:17,134 - Not downloading the file from http://ip-172-31-41-161.ec2.internal:8080/resources/DBConnectionVerification.jar, because /var/lib/ambari-agent/data/tmp/DBConnectionVerification.jar already exists
2015-07-13 13:01:17,135 - File['/usr/hdp/current/oozie-server/conf/hadoop-config.xml'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:17,136 - File['/usr/hdp/current/oozie-server/conf/oozie-default.xml'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:17,136 - Directory['/usr/hdp/current/oozie-server/conf/action-conf'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:17,137 - File['/usr/hdp/current/oozie-server/conf/action-conf/hive.xml'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:17,138 - File['/var/run/oozie/oozie.pid'] {'action': ['delete'], 'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
2015-07-13 13:01:17,244 - Directory['/usr/hdp/current/oozie-server//var/tmp/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,246 - Directory['/var/run/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,247 - Directory['/grid/0/log/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,247 - Directory['/var/tmp/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,248 - Directory['/hadoop/oozie/data'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,249 - Directory['/usr/hdp/current/oozie-server'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,249 - Directory['/usr/hdp/current/oozie-server/oozie-server/webapps'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,250 - Directory['/usr/hdp/current/oozie-server/oozie-server/conf'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
2015-07-13 13:01:17,251 - Directory['/usr/hdp/current/oozie-server/oozie-server'] {'owner': 'oozie', 'recursive': True, 'group': 'hadoop', 'mode': 0755, 'cd_access': 'a'}
2015-07-13 13:01:17,251 - Directory['/usr/hdp/current/oozie-server/libext'] {'recursive': True}
2015-07-13 13:01:17,252 - Execute['('tar', '-xvf', '/usr/hdp/current/oozie-server/oozie-sharelib.tar.gz', '-C', '/usr/hdp/current/oozie-server')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1' || test -f /usr/hdp/current/oozie-server/.hashcode && test -d /usr/hdp/current/oozie-server/share && [[ `cat /usr/hdp/current/oozie-server/.hashcode` == '046a880c90fcbbfea52bec80cb88dd8f' ]]", 'sudo': True}
2015-07-13 13:01:17,423 - Skipping Execute['('tar', '-xvf', '/usr/hdp/current/oozie-server/oozie-sharelib.tar.gz', '-C', '/usr/hdp/current/oozie-server')'] due to not_if
2015-07-13 13:01:17,425 - Execute['('cp', '/usr/share/HDP-oozie/ext-2.2.zip', '/usr/hdp/current/oozie-server/libext')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'sudo': True}
2015-07-13 13:01:17,631 - Execute['('chown', 'oozie:hadoop', '/usr/hdp/current/oozie-server/libext/ext-2.2.zip')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'sudo': True}
2015-07-13 13:01:17,814 - Execute['('chown', '-RL', 'oozie:hadoop', '/usr/hdp/current/oozie-server/oozie-server/conf')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'sudo': True}
2015-07-13 13:01:17,987 - File['/var/lib/ambari-agent/data/tmp/mysql-connector-java.jar'] {'content': DownloadSource('http://ip-172-31-41-161.ec2.internal:8080/resources//mysql-jdbc-driver.jar')}
2015-07-13 13:01:17,989 - Not downloading the file from http://ip-172-31-41-161.ec2.internal:8080/resources//mysql-jdbc-driver.jar, because /var/lib/ambari-agent/data/tmp/mysql-jdbc-driver.jar already exists
2015-07-13 13:01:17,993 - Execute['('cp', '--remove-destination', '/var/lib/ambari-agent/data/tmp/mysql-connector-java.jar', '/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar')'] {'path': ['/bin', '/usr/bin/'], 'sudo': True}
2015-07-13 13:01:18,099 - File['/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar'] {'owner': 'oozie', 'group': 'hadoop'}
2015-07-13 13:01:18,100 - Changing owner for /usr/hdp/current/oozie-server/libext/mysql-connector-java.jar from 0 to oozie
2015-07-13 13:01:18,100 - Changing group for /usr/hdp/current/oozie-server/libext/mysql-connector-java.jar from 0 to hadoop
2015-07-13 13:01:18,101 - Execute['ambari-sudo.sh cp /usr/hdp/current/falcon-client/oozie/ext/falcon-oozie-el-extension-*.jar /usr/hdp/current/oozie-server/libext'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
2015-07-13 13:01:18,350 - Execute['ambari-sudo.sh chown oozie:hadoop /usr/hdp/current/oozie-server/libext/falcon-oozie-el-extension-*.jar'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
2015-07-13 13:01:18,565 - Package['hadoop-lzo'] {}
2015-07-13 13:01:18,763 - Skipping installation of existing package hadoop-lzo
2015-07-13 13:01:18,764 - Package['lzo'] {}
2015-07-13 13:01:18,885 - Skipping installation of existing package lzo
2015-07-13 13:01:18,885 - Package['hadoop-lzo-native'] {}
2015-07-13 13:01:19,008 - Skipping installation of existing package hadoop-lzo-native
2015-07-13 13:01:19,008 - Package['hadooplzo_2_3_*'] {}
2015-07-13 13:01:19,154 - Skipping installation of existing package hadooplzo_2_3_*
2015-07-13 13:01:19,154 - Execute['ambari-sudo.sh cp /usr/hdp/current/hadoop-client/lib/hadoop-lzo*.jar /usr/hdp/current/oozie-server'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
2015-07-13 13:01:19,325 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] {'not_if': '...', 'user': 'oozie'}
2015-07-13 13:01:19,430 - Skipping Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] due to not_if
2015-07-13 13:01:19,431 - File['/usr/hdp/current/oozie-server/.hashcode'] {'content': '046a880c90fcbbfea52bec80cb88dd8f', 'mode': 0644}
2015-07-13 13:01:19,432 - File['/usr/hdp/current/oozie-server/.prepare_war_cmd'] {'content': 'cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war ', 'mode': 0644}
2015-07-13 13:01:19,432 - Directory['/usr/hdp/current/oozie-server/conf/action-conf/hive'] {'owner': 'oozie', 'group': 'hadoop', 'recursive': True}
2015-07-13 13:01:19,433 - XmlConfig['hive-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf/action-conf/hive', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
2015-07-13 13:01:19,447 - Generating config: /usr/hdp/current/oozie-server/conf/action-conf/hive/hive-site.xml
2015-07-13 13:01:19,447 - File['/usr/hdp/current/oozie-server/conf/action-conf/hive/hive-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2015-07-13 13:01:19,604 - Writing File['/usr/hdp/current/oozie-server/conf/action-conf/hive/hive-site.xml'] because contents don't match
2015-07-13 13:01:19,605 - XmlConfig['tez-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf/action-conf/hive', 'mode': 0664, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
2015-07-13 13:01:19,624 - Generating config: /usr/hdp/current/oozie-server/conf/action-conf/hive/tez-site.xml
2015-07-13 13:01:19,625 - File['/usr/hdp/current/oozie-server/conf/action-conf/hive/tez-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0664, 'encoding': 'UTF-8'}
2015-07-13 13:01:19,673 - Writing File['/usr/hdp/current/oozie-server/conf/action-conf/hive/tez-site.xml'] because contents don't match
2015-07-13 13:01:19,674 - Execute['('chown', '-R', 'oozie:hadoop', '/usr/hdp/current/oozie-server/oozie-server')'] {'sudo': True}
2015-07-13 13:01:19,798 - Execute['/usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://host/ooziedb' oozieuser [PROTECTED] com.mysql.jdbc.Driver'] {'tries': 5, 'try_sleep': 10}
2015-07-13 13:01:20,601 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/ooziedb.sh create -sqlfile oozie.sql -run'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'ignore_failures': True, 'user': 'oozie'}
2015-07-13 13:01:24,003 - checked_call['curl -sS -L -w '%{http_code}' -X GET 'http://ip-172-31-41-161.ec2.internal:50070/webhdfs/v1/user/oozie/share?op=GETFILESTATUS&user.name=oozie''] {'logoutput': None, 'user': 'oozie', 'stderr': -1, 'quiet': False}
2015-07-13 13:01:24,122 - checked_call returned (0, '{"RemoteException":{"exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException","message":"Permission denied: user=oozie, access=EXECUTE, inode=\\"/user/oozie/share\\":yarn:hadoop:drwx------"}}403', ''){code}


> Oozie Server start failed and RegionServers down after "Node Labels" enabled
> ----------------------------------------------------------------------------
>
>                 Key: AMBARI-12401
>                 URL: https://issues.apache.org/jira/browse/AMBARI-12401
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-web
>    Affects Versions: 2.1.0
>            Reporter: Sumit Mohanty
>            Assignee: Sumit Mohanty
>             Fix For: 2.1.0
>
>
> *scenario*
> # install ambari
> # deploy cluster
> # navigate YARN configs tab
> # set "Node Labels" property to "Enabled"
> # restart YARN and Oozie
> *expected* restart success
> *actual* *Oozie* restart failed and *RegionServers* randomly shut down
> {code}stderr:   /var/lib/ambari-agent/data/errors-227.txt
> Traceback (most recent call last):
>   File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 181, in <module>
>     OozieServer().execute()
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
>     method(env)
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 459, in restart
>     self.start(env)
>   File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 64, in start
>     oozie_service(action='start', rolling_restart=rolling_restart)
>   File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
>     return fn(*args, **kwargs)
>   File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py", line 102, in oozie_service
>     list_status = util.run_command(params.hdfs_share_dir, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 201, in run_command
>     raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ip-x-x-x-x.ec2.internal:50070/webhdfs/v1/user/oozie/share?op=GETFILESTATUS&user.name=oozie'' returned status_code=403. 
> {
>   "RemoteException": {
>     "exception": "AccessControlException", 
>     "javaClassName": "org.apache.hadoop.security.AccessControlException", 
>     "message": "Permission denied: user=oozie, access=EXECUTE, inode=\"/user/oozie/share\":yarn:hadoop:drwx------"
>   }
> }
> stdout:   /var/lib/ambari-agent/data/output-227.txt
> 2015-07-13 13:01:16,044 - Group['hadoop'] {'ignore_failures': False}
> 2015-07-13 13:01:16,060 - Group['users'] {'ignore_failures': False}
> 2015-07-13 13:01:16,060 - Group['knox'] {'ignore_failures': False}
> 2015-07-13 13:01:16,061 - Group['spark'] {'ignore_failures': False}
> 2015-07-13 13:01:16,061 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
> 2015-07-13 13:01:16,063 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,064 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
> 2015-07-13 13:01:16,065 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,066 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,068 - User['knox'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,069 - User['storm'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,070 - User['spark'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,071 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,073 - User['accumulo'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,074 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,075 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
> 2015-07-13 13:01:16,076 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,077 - User['falcon'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
> 2015-07-13 13:01:16,078 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,079 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,080 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,082 - User['ams'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,083 - User['atlas'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
> 2015-07-13 13:01:16,085 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2015-07-13 13:01:16,088 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
> 2015-07-13 13:01:16,153 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
> 2015-07-13 13:01:16,154 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
> 2015-07-13 13:01:16,156 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2015-07-13 13:01:16,158 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
> 2015-07-13 13:01:16,219 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
> 2015-07-13 13:01:16,220 - Group['hdfs'] {'ignore_failures': False}
> 2015-07-13 13:01:16,221 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']}
> 2015-07-13 13:01:16,222 - Directory['/etc/hadoop'] {'mode': 0755}
> 2015-07-13 13:01:16,252 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
> 2015-07-13 13:01:16,281 - Execute['('setenforce', '0')'] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
> 2015-07-13 13:01:16,415 - Directory['/grid/0/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
> 2015-07-13 13:01:16,417 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
> 2015-07-13 13:01:16,417 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
> 2015-07-13 13:01:16,423 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
> 2015-07-13 13:01:16,425 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
> 2015-07-13 13:01:16,426 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
> 2015-07-13 13:01:16,435 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
> 2015-07-13 13:01:16,436 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
> 2015-07-13 13:01:16,437 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
> 2015-07-13 13:01:16,445 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'group': 'hadoop'}
> 2015-07-13 13:01:16,445 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'mode': 0755}
> 2015-07-13 13:01:16,780 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh'] {'environment': {'OOZIE_CONFIG': '/usr/hdp/current/oozie-server/conf'}, 'only_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'user': 'oozie'}
> 2015-07-13 13:01:16,903 - Skipping Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh'] due to only_if
> 2015-07-13 13:01:16,903 - File['/var/run/oozie/oozie.pid'] {'action': ['delete']}
> 2015-07-13 13:01:16,910 - HdfsResource['/user/oozie'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://ip-x:8020', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'owner': 'oozie', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'mode': 0775}
> 2015-07-13 13:01:16,913 - checked_call['curl -sS -L -w '%{http_code}' -X GET 'http://ip-x:50070/webhdfs/v1/user/oozie?op=GETFILESTATUS&user.name=hdfs''] {'logoutput': None, 'user': 'hdfs', 'stderr': -1, 'quiet': False}
> 2015-07-13 13:01:17,046 - checked_call returned (0, '{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17109,"group":"hdfs","length":0,"modificationTime":1436785928420,"owner":"oozie","pathSuffix":"","permission":"775","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200', '')
> 2015-07-13 13:01:17,048 - HdfsResource['None'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://-x:8020', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf'}
> 2015-07-13 13:01:17,049 - Directory['/usr/hdp/current/oozie-server/conf'] {'owner': 'oozie', 'group': 'hadoop', 'recursive': True}
> 2015-07-13 13:01:17,050 - XmlConfig['oozie-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf', 'mode': 0664, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
> 2015-07-13 13:01:17,086 - Generating config: /usr/hdp/current/oozie-server/conf/oozie-site.xml
> 2015-07-13 13:01:17,086 - File['/usr/hdp/current/oozie-server/conf/oozie-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0664, 'encoding': 'UTF-8'}
> 2015-07-13 13:01:17,112 - Writing File['/usr/hdp/current/oozie-server/conf/oozie-site.xml'] because contents don't match
> 2015-07-13 13:01:17,122 - File['/usr/hdp/current/oozie-server/conf/oozie-env.sh'] {'content': InlineTemplate(...), 'owner': 'oozie'}
> 2015-07-13 13:01:17,123 - Writing File['/usr/hdp/current/oozie-server/conf/oozie-env.sh'] because contents don't match
> 2015-07-13 13:01:17,124 - File['/usr/hdp/current/oozie-server/conf/oozie-log4j.properties'] {'content': '...', 'owner': 'oozie', 'group': 'hadoop', 'mode': 0644}
> 2015-07-13 13:01:17,132 - File['/usr/hdp/current/oozie-server/conf/adminusers.txt'] {'content': Template('adminusers.txt.j2'), 'owner': 'oozie', 'group': 'hadoop', 'mode': 0644}
> 2015-07-13 13:01:17,134 - File['/usr/lib/ambari-agent/DBConnectionVerification.jar'] {'content': DownloadSource('http://ip--x:8080/resources/DBConnectionVerification.jar')}
> 2015-07-13 13:01:17,134 - Not downloading the file from http://ip-x:8080/resources/DBConnectionVerification.jar, because /var/lib/ambari-agent/data/tmp/DBConnectionVerification.jar already exists
> 2015-07-13 13:01:17,135 - File['/usr/hdp/current/oozie-server/conf/hadoop-config.xml'] {'owner': 'oozie', 'group': 'hadoop'}
> 2015-07-13 13:01:17,136 - File['/usr/hdp/current/oozie-server/conf/oozie-default.xml'] {'owner': 'oozie', 'group': 'hadoop'}
> 2015-07-13 13:01:17,136 - Directory['/usr/hdp/current/oozie-server/conf/action-conf'] {'owner': 'oozie', 'group': 'hadoop'}
> 2015-07-13 13:01:17,137 - File['/usr/hdp/current/oozie-server/conf/action-conf/hive.xml'] {'owner': 'oozie', 'group': 'hadoop'}
> 2015-07-13 13:01:17,138 - File['/var/run/oozie/oozie.pid'] {'action': ['delete'], 'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
> 2015-07-13 13:01:17,244 - Directory['/usr/hdp/current/oozie-server//var/tmp/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
> 2015-07-13 13:01:17,246 - Directory['/var/run/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
> 2015-07-13 13:01:17,247 - Directory['/grid/0/log/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
> 2015-07-13 13:01:17,247 - Directory['/var/tmp/oozie'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
> 2015-07-13 13:01:17,248 - Directory['/hadoop/oozie/data'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
> 2015-07-13 13:01:17,249 - Directory['/usr/hdp/current/oozie-server'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
> 2015-07-13 13:01:17,249 - Directory['/usr/hdp/current/oozie-server/oozie-server/webapps'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
> 2015-07-13 13:01:17,250 - Directory['/usr/hdp/current/oozie-server/oozie-server/conf'] {'owner': 'oozie', 'cd_access': 'a', 'group': 'hadoop', 'recursive': True, 'mode': 0755}
> 2015-07-13 13:01:17,251 - Directory['/usr/hdp/current/oozie-server/oozie-server'] {'owner': 'oozie', 'recursive': True, 'group': 'hadoop', 'mode': 0755, 'cd_access': 'a'}
> 2015-07-13 13:01:17,251 - Directory['/usr/hdp/current/oozie-server/libext'] {'recursive': True}
> 2015-07-13 13:01:17,252 - Execute['('tar', '-xvf', '/usr/hdp/current/oozie-server/oozie-sharelib.tar.gz', '-C', '/usr/hdp/current/oozie-server')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1' || test -f /usr/hdp/current/oozie-server/.hashcode && test -d /usr/hdp/current/oozie-server/share && [[ `cat /usr/hdp/current/oozie-server/.hashcode` == '046a880c90fcbbfea52bec80cb88dd8f' ]]", 'sudo': True}
> 2015-07-13 13:01:17,423 - Skipping Execute['('tar', '-xvf', '/usr/hdp/current/oozie-server/oozie-sharelib.tar.gz', '-C', '/usr/hdp/current/oozie-server')'] due to not_if
> 2015-07-13 13:01:17,425 - Execute['('cp', '/usr/share/HDP-oozie/ext-2.2.zip', '/usr/hdp/current/oozie-server/libext')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'sudo': True}
> 2015-07-13 13:01:17,631 - Execute['('chown', 'oozie:hadoop', '/usr/hdp/current/oozie-server/libext/ext-2.2.zip')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'sudo': True}
> 2015-07-13 13:01:17,814 - Execute['('chown', '-RL', 'oozie:hadoop', '/usr/hdp/current/oozie-server/oozie-server/conf')'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'sudo': True}
> 2015-07-13 13:01:17,987 - File['/var/lib/ambari-agent/data/tmp/mysql-connector-java.jar'] {'content': DownloadSource('http://ip-x:8080/resources//mysql-jdbc-driver.jar')}
> 2015-07-13 13:01:17,989 - Not downloading the file from http://ip-x:8080/resources//mysql-jdbc-driver.jar, because /var/lib/ambari-agent/data/tmp/mysql-jdbc-driver.jar already exists
> 2015-07-13 13:01:17,993 - Execute['('cp', '--remove-destination', '/var/lib/ambari-agent/data/tmp/mysql-connector-java.jar', '/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar')'] {'path': ['/bin', '/usr/bin/'], 'sudo': True}
> 2015-07-13 13:01:18,099 - File['/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar'] {'owner': 'oozie', 'group': 'hadoop'}
> 2015-07-13 13:01:18,100 - Changing owner for /usr/hdp/current/oozie-server/libext/mysql-connector-java.jar from 0 to oozie
> 2015-07-13 13:01:18,100 - Changing group for /usr/hdp/current/oozie-server/libext/mysql-connector-java.jar from 0 to hadoop
> 2015-07-13 13:01:18,101 - Execute['ambari-sudo.sh cp /usr/hdp/current/falcon-client/oozie/ext/falcon-oozie-el-extension-*.jar /usr/hdp/current/oozie-server/libext'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
> 2015-07-13 13:01:18,350 - Execute['ambari-sudo.sh chown oozie:hadoop /usr/hdp/current/oozie-server/libext/falcon-oozie-el-extension-*.jar'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
> 2015-07-13 13:01:18,565 - Package['hadoop-lzo'] {}
> 2015-07-13 13:01:18,763 - Skipping installation of existing package hadoop-lzo
> 2015-07-13 13:01:18,764 - Package['lzo'] {}
> 2015-07-13 13:01:18,885 - Skipping installation of existing package lzo
> 2015-07-13 13:01:18,885 - Package['hadoop-lzo-native'] {}
> 2015-07-13 13:01:19,008 - Skipping installation of existing package hadoop-lzo-native
> 2015-07-13 13:01:19,008 - Package['hadooplzo_2_3_*'] {}
> 2015-07-13 13:01:19,154 - Skipping installation of existing package hadooplzo_2_3_*
> 2015-07-13 13:01:19,154 - Execute['ambari-sudo.sh cp /usr/hdp/current/hadoop-client/lib/hadoop-lzo*.jar /usr/hdp/current/oozie-server'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'"}
> 2015-07-13 13:01:19,325 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] {'not_if': '...', 'user': 'oozie'}
> 2015-07-13 13:01:19,430 - Skipping Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] due to not_if
> 2015-07-13 13:01:19,431 - File['/usr/hdp/current/oozie-server/.hashcode'] {'content': '046a880c90fcbbfea52bec80cb88dd8f', 'mode': 0644}
> 2015-07-13 13:01:19,432 - File['/usr/hdp/current/oozie-server/.prepare_war_cmd'] {'content': 'cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war ', 'mode': 0644}
> 2015-07-13 13:01:19,432 - Directory['/usr/hdp/current/oozie-server/conf/action-conf/hive'] {'owner': 'oozie', 'group': 'hadoop', 'recursive': True}
> 2015-07-13 13:01:19,433 - XmlConfig['hive-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf/action-conf/hive', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
> 2015-07-13 13:01:19,447 - Generating config: /usr/hdp/current/oozie-server/conf/action-conf/hive/hive-site.xml
> 2015-07-13 13:01:19,447 - File['/usr/hdp/current/oozie-server/conf/action-conf/hive/hive-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
> 2015-07-13 13:01:19,604 - Writing File['/usr/hdp/current/oozie-server/conf/action-conf/hive/hive-site.xml'] because contents don't match
> 2015-07-13 13:01:19,605 - XmlConfig['tez-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf/action-conf/hive', 'mode': 0664, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
> 2015-07-13 13:01:19,624 - Generating config: /usr/hdp/current/oozie-server/conf/action-conf/hive/tez-site.xml
> 2015-07-13 13:01:19,625 - File['/usr/hdp/current/oozie-server/conf/action-conf/hive/tez-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0664, 'encoding': 'UTF-8'}
> 2015-07-13 13:01:19,673 - Writing File['/usr/hdp/current/oozie-server/conf/action-conf/hive/tez-site.xml'] because contents don't match
> 2015-07-13 13:01:19,674 - Execute['('chown', '-R', 'oozie:hadoop', '/usr/hdp/current/oozie-server/oozie-server')'] {'sudo': True}
> 2015-07-13 13:01:19,798 - Execute['/usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://host/ooziedb' oozieuser [PROTECTED] com.mysql.jdbc.Driver'] {'tries': 5, 'try_sleep': 10}
> 2015-07-13 13:01:20,601 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/ooziedb.sh create -sqlfile oozie.sql -run'] {'not_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'ignore_failures': True, 'user': 'oozie'}
> 2015-07-13 13:01:24,003 - checked_call['curl -sS -L -w '%{http_code}' -X GET 'http://ip-x:50070/webhdfs/v1/user/oozie/share?op=GETFILESTATUS&user.name=oozie''] {'logoutput': None, 'user': 'oozie', 'stderr': -1, 'quiet': False}
> 2015-07-13 13:01:24,122 - checked_call returned (0, '{"RemoteException":{"exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException","message":"Permission denied: user=oozie, access=EXECUTE, inode=\\"/user/oozie/share\\":yarn:hadoop:drwx------"}}403', ''){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message