ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shaik M <munna.had...@gmail.com>
Subject Re: Ambari Unable to start Hive server 2 after enabling security
Date Mon, 04 May 2015 13:54:48 GMT
Hi,


*I am getting following exception in Hiveserver2 log file:*

2015-05-04 13:45:23,542 WARN  [main]: ipc.Client (Client.java:run(676)) -
Exception encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]
2015-05-04 13:45:23,543 INFO  [main]: retry.RetryInvocationHandler
(RetryInvocationHandler.java:invoke(140)) - Exception while invoking
getFileInfo of class ClientNamenodeProtocolTranslatorPB over
sv2lxbdp2mst05.corp.host.com/10.192.149.187:8020 after 8 fail over
attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)];

Thanks,
Shaik

(If it is not right place to post this query, please help me to know the
correct group)

On 4 May 2015 at 18:55, Shaik M <munna.hadoop@gmail.com> wrote:

> Hi,
>
> I am using Ambari 1.7 and HDP 2.2.4. I have enabled security in this
> cluster.
>
> After enabling the security Ambari unable start the Hive server 2.
>
> I have verified keytabs and it's working fine. Please find the below
> ambari string process log below. not shown any output in standerr window.
>
> please let me know the how to resolve this issue.
>
> Thanks,
> Shaik M
>
> stdout:   /var/lib/ambari-agent/data/output-1832.txt
>
> 2015-05-04 10:47:25,616 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/;
    curl -kf -x "" --retry 10     http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip
-o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment':
..., 'not_if': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip',
'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
> 2015-05-04 10:47:25,646 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/;
    curl -kf -x "" --retry 10     http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip
-o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if
> 2015-05-04 10:47:25,648 - Execute['rm -f local_policy.jar; rm -f US_export_policy.jar;
unzip -o -j -q /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip']
{'path': ['/bin/', '/usr/bin'], 'only_if': 'test -e /usr/jdk64/jdk1.7.0_67/jre/lib/security
&& test -f /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip',
'cwd': '/usr/jdk64/jdk1.7.0_67/jre/lib/security'}
> 2015-05-04 10:47:25,718 - Group['hadoop'] {'ignore_failures': False}
> 2015-05-04 10:47:25,720 - Modifying group hadoop
> 2015-05-04 10:47:25,784 - Group['nobody'] {'ignore_failures': False}
> 2015-05-04 10:47:25,785 - Modifying group nobody
> 2015-05-04 10:47:25,839 - Group['users'] {'ignore_failures': False}
> 2015-05-04 10:47:25,840 - Modifying group users
> 2015-05-04 10:47:25,886 - Group['nagios'] {'ignore_failures': False}
> 2015-05-04 10:47:25,887 - Modifying group nagios
> 2015-05-04 10:47:25,936 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False,
'groups': [u'nobody']}
> 2015-05-04 10:47:25,937 - Modifying user nobody
> 2015-05-04 10:47:26,012 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups':
[u'users']}
> 2015-05-04 10:47:26,013 - Modifying user oozie
> 2015-05-04 10:47:26,063 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups':
[u'hadoop']}
> 2015-05-04 10:47:26,064 - Modifying user hive
> 2015-05-04 10:47:26,098 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False,
'groups': [u'hadoop']}
> 2015-05-04 10:47:26,099 - Modifying user mapred
> 2015-05-04 10:47:26,131 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False,
'groups': [u'hadoop']}
> 2015-05-04 10:47:26,132 - Modifying user nagios
> 2015-05-04 10:47:26,165 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False,
'groups': [u'users']}
> 2015-05-04 10:47:26,166 - Modifying user ambari-qa
> 2015-05-04 10:47:26,199 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False,
'groups': [u'hadoop']}
> 2015-05-04 10:47:26,199 - Modifying user zookeeper
> 2015-05-04 10:47:26,232 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups':
[u'users']}
> 2015-05-04 10:47:26,233 - Modifying user tez
> 2015-05-04 10:47:26,265 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups':
[u'hadoop']}
> 2015-05-04 10:47:26,266 - Modifying user hdfs
> 2015-05-04 10:47:26,298 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups':
[u'hadoop']}
> 2015-05-04 10:47:26,299 - Modifying user sqoop
> 2015-05-04 10:47:26,332 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups':
[u'hadoop']}
> 2015-05-04 10:47:26,332 - Modifying user hcat
> 2015-05-04 10:47:26,365 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups':
[u'hadoop']}
> 2015-05-04 10:47:26,366 - Modifying user yarn
> 2015-05-04 10:47:26,399 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content':
StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2015-05-04 10:47:26,401 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa
/tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa
2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
> 2015-05-04 10:47:26,431 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh
ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa
2>/dev/null'] due to not_if
> 2015-05-04 10:47:26,432 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group':
'root', 'recursive': True}
> 2015-05-04 10:47:26,433 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf',
'to': '/etc/hadoop/conf.empty'}
> 2015-05-04 10:47:26,462 - Skipping Link['/etc/hadoop/conf'] due to not_if
> 2015-05-04 10:47:26,491 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...),
'owner': 'root'}
> 2015-05-04 10:47:26,516 - Execute['/bin/echo 0 > /selinux/enforce'] {'only_if': 'test
-f /selinux/enforce'}
> 2015-05-04 10:47:26,577 - Directory['/var/log/hadoop'] {'owner': 'root', 'group': 'hadoop',
'mode': 0775, 'recursive': True}
> 2015-05-04 10:47:26,578 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root',
'recursive': True}
> 2015-05-04 10:47:26,579 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive':
True}
> 2015-05-04 10:47:26,590 - File['/etc/hadoop/conf/commons-logging.properties'] {'content':
Template('commons-logging.properties.j2'), 'owner': 'root'}
> 2015-05-04 10:47:26,595 - File['/etc/hadoop/conf/health_check'] {'content': Template('health_check-v2.j2'),
'owner': 'root'}
> 2015-05-04 10:47:26,596 - File['/etc/hadoop/conf/log4j.properties'] {'content': '...',
'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
> 2015-05-04 10:47:26,608 - File['/etc/hadoop/conf/hadoop-metrics2.properties'] {'content':
Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
> 2015-05-04 10:47:26,609 - File['/etc/hadoop/conf/task-log4j.properties'] {'content':
StaticFile('task-log4j.properties'), 'mode': 0755}
> 2015-05-04 10:47:26,956 - Execute['kill `cat /var/run/hive/hive-server.pid` >/dev/null
2>&1 && rm -f /var/run/hive/hive-server.pid'] {'not_if': '! (ls /var/run/hive/hive-server.pid
>/dev/null 2>&1 && ps `cat /var/run/hive/hive-server.pid` >/dev/null
2>&1)'}
> 2015-05-04 10:47:27,068 - HdfsDirectory['/apps/hive/warehouse'] {'security_enabled':
True, 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf',
'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 0777, 'owner': 'hive',
'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']}
> 2015-05-04 10:47:27,069 - HdfsDirectory['/user/hive'] {'security_enabled': True, 'keytab':
'/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf', 'hdfs_user':
'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 0700, 'owner': 'hive', 'bin_dir': '/usr/hdp/current/hadoop-client/bin',
'action': ['create_delayed']}
> 2015-05-04 10:47:27,070 - HdfsDirectory['None'] {'security_enabled': True, 'keytab':
'/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf', 'hdfs_user':
'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'action': ['create'], 'bin_dir': '/usr/hdp/current/hadoop-client/bin'}
> 2015-05-04 10:47:27,073 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab
hdfs'] {'user': 'hdfs'}
> 2015-05-04 10:47:27,850 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q
hadoop | grep -q "hadoop-1" || echo "-p"` /apps/hive/warehouse /user/hive && hadoop
--config /etc/hadoop/conf fs -chmod  777 /apps/hive/warehouse && hadoop --config /etc/hadoop/conf
fs -chmod  700 /user/hive && hadoop --config /etc/hadoop/conf fs -chown  hive /apps/hive/warehouse
/user/hive'] {'not_if': "su - hdfs -c 'export PATH=$PATH:/usr/hdp/current/hadoop-client/bin
; hadoop --config /etc/hadoop/conf fs -ls /apps/hive/warehouse /user/hive'", 'user': 'hdfs',
'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2015-05-04 10:47:31,133 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /apps/hive/warehouse /user/hive &&
hadoop --config /etc/hadoop/conf fs -chmod  777 /apps/hive/warehouse && hadoop --config
/etc/hadoop/conf fs -chmod  700 /user/hive && hadoop --config /etc/hadoop/conf fs
-chown  hive /apps/hive/warehouse /user/hive'] due to not_if
> 2015-05-04 10:47:31,134 - Directory['/etc/hive/conf.server'] {'owner': 'hive', 'group':
'hadoop', 'recursive': True}
> 2015-05-04 10:47:31,135 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir':
'/etc/hive/conf.server', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hive', 'configurations':
...}
> 2015-05-04 10:47:31,168 - Generating config: /etc/hive/conf.server/mapred-site.xml
> 2015-05-04 10:47:31,169 - File['/etc/hive/conf.server/mapred-site.xml'] {'owner': 'hive',
'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
> 2015-05-04 10:47:31,173 - Writing File['/etc/hive/conf.server/mapred-site.xml'] because
contents don't match
> 2015-05-04 10:47:31,174 - File['/etc/hive/conf.server/hive-default.xml.template'] {'owner':
'hive', 'group': 'hadoop'}
> 2015-05-04 10:47:31,175 - File['/etc/hive/conf.server/hive-env.sh.template'] {'owner':
'hive', 'group': 'hadoop'}
> 2015-05-04 10:47:31,177 - File['/etc/hive/conf.server/hive-exec-log4j.properties'] {'content':
'...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
> 2015-05-04 10:47:31,178 - File['/etc/hive/conf.server/hive-log4j.properties'] {'content':
'...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
> 2015-05-04 10:47:31,179 - Directory['/etc/hive/conf'] {'owner': 'hive', 'group': 'hadoop',
'recursive': True}
> 2015-05-04 10:47:31,180 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir':
'/etc/hive/conf', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hive', 'configurations':
...}
> 2015-05-04 10:47:31,202 - Generating config: /etc/hive/conf/mapred-site.xml
> 2015-05-04 10:47:31,203 - File['/etc/hive/conf/mapred-site.xml'] {'owner': 'hive', 'content':
InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
> 2015-05-04 10:47:31,207 - Writing File['/etc/hive/conf/mapred-site.xml'] because contents
don't match
> 2015-05-04 10:47:31,208 - File['/etc/hive/conf/hive-default.xml.template'] {'owner':
'hive', 'group': 'hadoop'}
> 2015-05-04 10:47:31,209 - File['/etc/hive/conf/hive-env.sh.template'] {'owner': 'hive',
'group': 'hadoop'}
> 2015-05-04 10:47:31,211 - File['/etc/hive/conf/hive-exec-log4j.properties'] {'content':
'...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
> 2015-05-04 10:47:31,212 - File['/etc/hive/conf/hive-log4j.properties'] {'content': '...',
'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
> 2015-05-04 10:47:31,213 - XmlConfig['hive-site.xml'] {'group': 'hadoop', 'conf_dir':
'/etc/hive/conf.server', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hive', 'configurations':
...}
> 2015-05-04 10:47:31,236 - Generating config: /etc/hive/conf.server/hive-site.xml
> 2015-05-04 10:47:31,237 - File['/etc/hive/conf.server/hive-site.xml'] {'owner': 'hive',
'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
> 2015-05-04 10:47:31,245 - Writing File['/etc/hive/conf.server/hive-site.xml'] because
contents don't match
> 2015-05-04 10:47:31,251 - File['/etc/hive/conf.server/hive-env.sh'] {'content': InlineTemplate(...),
'owner': 'hive', 'group': 'hadoop'}
> 2015-05-04 10:47:31,253 - Execute['hive mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/
; rm -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp /usr/share/java/mysql-connector-java.jar
/usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] {'environment': ..., 'path': ['/bin',
'/usr/bin/'], 'creates': '/usr/hdp/current/hive-client/lib/mysql-connector-java.jar', 'not_if':
'test -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'}
> 2015-05-04 10:47:31,282 - Skipping Execute['hive mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/
; rm -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp /usr/share/java/mysql-connector-java.jar
/usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] due to not_if
> 2015-05-04 10:47:31,285 - Execute['/bin/sh -c 'cd /usr/lib/ambari-agent/ && curl
-kf -x "" --retry 5 http://sv2lxbdp2mst05.corp.equinix.com:8080/resources/DBConnectionVerification.jar
-o DBConnectionVerification.jar''] {'environment': ..., 'not_if': '[ -f DBConnectionVerification.jar]'}
> 2015-05-04 10:47:31,370 - File['/var/lib/ambari-agent/data/tmp/start_hiveserver2_script']
{'content': Template('startHiveserver2.sh.j2'), 'mode': 0755}
> 2015-05-04 10:47:31,372 - Directory['/var/run/hive'] {'owner': 'hive', 'group': 'hadoop',
'mode': 0755, 'recursive': True}
> 2015-05-04 10:47:31,372 - Directory['/var/log/hive'] {'owner': 'hive', 'group': 'hadoop',
'mode': 0755, 'recursive': True}
> 2015-05-04 10:47:31,373 - Directory['/var/lib/hive'] {'owner': 'hive', 'group': 'hadoop',
'mode': 0755, 'recursive': True}
> 2015-05-04 10:47:31,453 - Could not verify HDP version by calling '/usr/bin/hdp-select
versions > /tmp/tmpTrvIN3'. Return Code: 0, Output: 2.2.4.2-2
> .
> 2015-05-04 10:47:31,531 - Could not verify HDP version by calling '/usr/bin/hdp-select
versions > /tmp/tmpHFa97f'. Return Code: 0, Output: 2.2.4.2-2
> .
> 2015-05-04 10:47:31,600 - Execute['env JAVA_HOME=/usr/jdk64/jdk1.7.0_67 /var/lib/ambari-agent/data/tmp/start_hiveserver2_script
/var/log/hive/hive-server2.out /var/log/hive/hive-server2.log /var/run/hive/hive-server.pid
/etc/hive/conf.server /var/log/hive'] {'environment': ..., 'not_if': 'ls /var/run/hive/hive-server.pid
>/dev/null 2>&1 && ps `cat /var/run/hive/hive-server.pid` >/dev/null
2>&1', 'user': 'hive', 'path': ['/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin']}
> 2015-05-04 10:47:31,693 - Execute['/usr/jdk64/jdk1.7.0_67/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar
org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://sv2lxbdp2mst04.corp.equinix.com/hive?createDatabaseIfNotExist=true'
hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'],
'tries': 5, 'try_sleep': 10}
>
>

Mime
View raw message