ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shaik M <munna.had...@gmail.com>
Subject Re: Ambari Unable to start Hive server 2 after enabling security
Date Tue, 05 May 2015 09:15:49 GMT
Ambari Team,

can you please check, why it is required to do manual "kinit" for
HiveServer2 startup?



On 4 May 2015 at 22:24, Shaik M <munna.hadoop@gmail.com> wrote:

> Hi,
>
> After verifying the Amabri agent log, I logged in as "hive"  user and I
> ran below command.
>
> /usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs
>
> after that I have started "hiveserver2" service from Amabri and it started
> the service. Now, I can able to connect to beeline client.
>
> I have re-verified, with destroying the key and the hiveserver2 service
> went down after destroying the key.
>
> Look like Ambari having some issue to initiate the HiveServer2 keytabs.
>
> -Shaik
>
>
>
>
>
>
>
>
>
> On 4 May 2015 at 21:54, Shaik M <munna.hadoop@gmail.com> wrote:
>
>> Hi,
>>
>>
>> *I am getting following exception in Hiveserver2 log file:*
>>
>> 2015-05-04 13:45:23,542 WARN  [main]: ipc.Client (Client.java:run(676)) -
>> Exception encountered while connecting to the server :
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Failed to
>> find any Kerberos tgt)]
>> 2015-05-04 13:45:23,543 INFO  [main]: retry.RetryInvocationHandler
>> (RetryInvocationHandler.java:invoke(140)) - Exception while invoking
>> getFileInfo of class ClientNamenodeProtocolTranslatorPB over
>> sv2lxbdp2mst05.corp.host.com/10.192.149.187:8020 after 8 fail over
>> attempts. Trying to fail over immediately.
>> java.io.IOException: Failed on local exception: java.io.IOException:
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Failed to
>> find any Kerberos tgt)];
>>
>> Thanks,
>> Shaik
>>
>> (If it is not right place to post this query, please help me to know the
>> correct group)
>>
>> On 4 May 2015 at 18:55, Shaik M <munna.hadoop@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am using Ambari 1.7 and HDP 2.2.4. I have enabled security in this
>>> cluster.
>>>
>>> After enabling the security Ambari unable start the Hive server 2.
>>>
>>> I have verified keytabs and it's working fine. Please find the below
>>> ambari string process log below. not shown any output in standerr window.
>>>
>>> please let me know the how to resolve this issue.
>>>
>>> Thanks,
>>> Shaik M
>>>
>>> stdout:   /var/lib/ambari-agent/data/output-1832.txt
>>>
>>> 2015-05-04 10:47:25,616 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/;
    curl -kf -x "" --retry 10     http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip
-o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment':
..., 'not_if': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip',
'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
>>> 2015-05-04 10:47:25,646 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/;
    curl -kf -x "" --retry 10     http://sv2lxbdp2mst05.corp.equinix.com:8080/resources//UnlimitedJCEPolicyJDK7.zip
-o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if
>>> 2015-05-04 10:47:25,648 - Execute['rm -f local_policy.jar; rm -f US_export_policy.jar;
unzip -o -j -q /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip']
{'path': ['/bin/', '/usr/bin'], 'only_if': 'test -e /usr/jdk64/jdk1.7.0_67/jre/lib/security
&& test -f /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip',
'cwd': '/usr/jdk64/jdk1.7.0_67/jre/lib/security'}
>>> 2015-05-04 10:47:25,718 - Group['hadoop'] {'ignore_failures': False}
>>> 2015-05-04 10:47:25,720 - Modifying group hadoop
>>> 2015-05-04 10:47:25,784 - Group['nobody'] {'ignore_failures': False}
>>> 2015-05-04 10:47:25,785 - Modifying group nobody
>>> 2015-05-04 10:47:25,839 - Group['users'] {'ignore_failures': False}
>>> 2015-05-04 10:47:25,840 - Modifying group users
>>> 2015-05-04 10:47:25,886 - Group['nagios'] {'ignore_failures': False}
>>> 2015-05-04 10:47:25,887 - Modifying group nagios
>>> 2015-05-04 10:47:25,936 - User['nobody'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'nobody']}
>>> 2015-05-04 10:47:25,937 - Modifying user nobody
>>> 2015-05-04 10:47:26,012 - User['oozie'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'users']}
>>> 2015-05-04 10:47:26,013 - Modifying user oozie
>>> 2015-05-04 10:47:26,063 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False,
'groups': [u'hadoop']}
>>> 2015-05-04 10:47:26,064 - Modifying user hive
>>> 2015-05-04 10:47:26,098 - User['mapred'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'hadoop']}
>>> 2015-05-04 10:47:26,099 - Modifying user mapred
>>> 2015-05-04 10:47:26,131 - User['nagios'] {'gid': 'nagios', 'ignore_failures':
False, 'groups': [u'hadoop']}
>>> 2015-05-04 10:47:26,132 - Modifying user nagios
>>> 2015-05-04 10:47:26,165 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'users']}
>>> 2015-05-04 10:47:26,166 - Modifying user ambari-qa
>>> 2015-05-04 10:47:26,199 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'hadoop']}
>>> 2015-05-04 10:47:26,199 - Modifying user zookeeper
>>> 2015-05-04 10:47:26,232 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False,
'groups': [u'users']}
>>> 2015-05-04 10:47:26,233 - Modifying user tez
>>> 2015-05-04 10:47:26,265 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False,
'groups': [u'hadoop']}
>>> 2015-05-04 10:47:26,266 - Modifying user hdfs
>>> 2015-05-04 10:47:26,298 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures':
False, 'groups': [u'hadoop']}
>>> 2015-05-04 10:47:26,299 - Modifying user sqoop
>>> 2015-05-04 10:47:26,332 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False,
'groups': [u'hadoop']}
>>> 2015-05-04 10:47:26,332 - Modifying user hcat
>>> 2015-05-04 10:47:26,365 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False,
'groups': [u'hadoop']}
>>> 2015-05-04 10:47:26,366 - Modifying user yarn
>>> 2015-05-04 10:47:26,399 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh']
{'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
>>> 2015-05-04 10:47:26,401 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh
ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa
2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
>>> 2015-05-04 10:47:26,431 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh
ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa
2>/dev/null'] due to not_if
>>> 2015-05-04 10:47:26,432 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root',
'group': 'root', 'recursive': True}
>>> 2015-05-04 10:47:26,433 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf',
'to': '/etc/hadoop/conf.empty'}
>>> 2015-05-04 10:47:26,462 - Skipping Link['/etc/hadoop/conf'] due to not_if
>>> 2015-05-04 10:47:26,491 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content':
InlineTemplate(...), 'owner': 'root'}
>>> 2015-05-04 10:47:26,516 - Execute['/bin/echo 0 > /selinux/enforce'] {'only_if':
'test -f /selinux/enforce'}
>>> 2015-05-04 10:47:26,577 - Directory['/var/log/hadoop'] {'owner': 'root', 'group':
'hadoop', 'mode': 0775, 'recursive': True}
>>> 2015-05-04 10:47:26,578 - Directory['/var/run/hadoop'] {'owner': 'root', 'group':
'root', 'recursive': True}
>>> 2015-05-04 10:47:26,579 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive':
True}
>>> 2015-05-04 10:47:26,590 - File['/etc/hadoop/conf/commons-logging.properties']
{'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
>>> 2015-05-04 10:47:26,595 - File['/etc/hadoop/conf/health_check'] {'content': Template('health_check-v2.j2'),
'owner': 'root'}
>>> 2015-05-04 10:47:26,596 - File['/etc/hadoop/conf/log4j.properties'] {'content':
'...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
>>> 2015-05-04 10:47:26,608 - File['/etc/hadoop/conf/hadoop-metrics2.properties']
{'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
>>> 2015-05-04 10:47:26,609 - File['/etc/hadoop/conf/task-log4j.properties'] {'content':
StaticFile('task-log4j.properties'), 'mode': 0755}
>>> 2015-05-04 10:47:26,956 - Execute['kill `cat /var/run/hive/hive-server.pid` >/dev/null
2>&1 && rm -f /var/run/hive/hive-server.pid'] {'not_if': '! (ls /var/run/hive/hive-server.pid
>/dev/null 2>&1 && ps `cat /var/run/hive/hive-server.pid` >/dev/null
2>&1)'}
>>> 2015-05-04 10:47:27,068 - HdfsDirectory['/apps/hive/warehouse'] {'security_enabled':
True, 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf',
'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 0777, 'owner': 'hive',
'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']}
>>> 2015-05-04 10:47:27,069 - HdfsDirectory['/user/hive'] {'security_enabled': True,
'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf', 'hdfs_user':
'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 0700, 'owner': 'hive', 'bin_dir': '/usr/hdp/current/hadoop-client/bin',
'action': ['create_delayed']}
>>> 2015-05-04 10:47:27,070 - HdfsDirectory['None'] {'security_enabled': True, 'keytab':
'/etc/security/keytabs/hdfs.headless.keytab', 'conf_dir': '/etc/hadoop/conf', 'hdfs_user':
'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'action': ['create'], 'bin_dir': '/usr/hdp/current/hadoop-client/bin'}
>>> 2015-05-04 10:47:27,073 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab
hdfs'] {'user': 'hdfs'}
>>> 2015-05-04 10:47:27,850 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir
`rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /apps/hive/warehouse /user/hive &&
hadoop --config /etc/hadoop/conf fs -chmod  777 /apps/hive/warehouse && hadoop --config
/etc/hadoop/conf fs -chmod  700 /user/hive && hadoop --config /etc/hadoop/conf fs
-chown  hive /apps/hive/warehouse /user/hive'] {'not_if': "su - hdfs -c 'export PATH=$PATH:/usr/hdp/current/hadoop-client/bin
; hadoop --config /etc/hadoop/conf fs -ls /apps/hive/warehouse /user/hive'", 'user': 'hdfs',
'path': ['/usr/hdp/current/hadoop-client/bin']}
>>> 2015-05-04 10:47:31,133 - Skipping Execute['hadoop --config /etc/hadoop/conf
fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /apps/hive/warehouse /user/hive
&& hadoop --config /etc/hadoop/conf fs -chmod  777 /apps/hive/warehouse &&
hadoop --config /etc/hadoop/conf fs -chmod  700 /user/hive && hadoop --config /etc/hadoop/conf
fs -chown  hive /apps/hive/warehouse /user/hive'] due to not_if
>>> 2015-05-04 10:47:31,134 - Directory['/etc/hive/conf.server'] {'owner': 'hive',
'group': 'hadoop', 'recursive': True}
>>> 2015-05-04 10:47:31,135 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir':
'/etc/hive/conf.server', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hive', 'configurations':
...}
>>> 2015-05-04 10:47:31,168 - Generating config: /etc/hive/conf.server/mapred-site.xml
>>> 2015-05-04 10:47:31,169 - File['/etc/hive/conf.server/mapred-site.xml'] {'owner':
'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
>>> 2015-05-04 10:47:31,173 - Writing File['/etc/hive/conf.server/mapred-site.xml']
because contents don't match
>>> 2015-05-04 10:47:31,174 - File['/etc/hive/conf.server/hive-default.xml.template']
{'owner': 'hive', 'group': 'hadoop'}
>>> 2015-05-04 10:47:31,175 - File['/etc/hive/conf.server/hive-env.sh.template']
{'owner': 'hive', 'group': 'hadoop'}
>>> 2015-05-04 10:47:31,177 - File['/etc/hive/conf.server/hive-exec-log4j.properties']
{'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
>>> 2015-05-04 10:47:31,178 - File['/etc/hive/conf.server/hive-log4j.properties']
{'content': '...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
>>> 2015-05-04 10:47:31,179 - Directory['/etc/hive/conf'] {'owner': 'hive', 'group':
'hadoop', 'recursive': True}
>>> 2015-05-04 10:47:31,180 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir':
'/etc/hive/conf', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hive', 'configurations':
...}
>>> 2015-05-04 10:47:31,202 - Generating config: /etc/hive/conf/mapred-site.xml
>>> 2015-05-04 10:47:31,203 - File['/etc/hive/conf/mapred-site.xml'] {'owner': 'hive',
'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
>>> 2015-05-04 10:47:31,207 - Writing File['/etc/hive/conf/mapred-site.xml'] because
contents don't match
>>> 2015-05-04 10:47:31,208 - File['/etc/hive/conf/hive-default.xml.template'] {'owner':
'hive', 'group': 'hadoop'}
>>> 2015-05-04 10:47:31,209 - File['/etc/hive/conf/hive-env.sh.template'] {'owner':
'hive', 'group': 'hadoop'}
>>> 2015-05-04 10:47:31,211 - File['/etc/hive/conf/hive-exec-log4j.properties'] {'content':
'...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
>>> 2015-05-04 10:47:31,212 - File['/etc/hive/conf/hive-log4j.properties'] {'content':
'...', 'owner': 'hive', 'group': 'hadoop', 'mode': 0644}
>>> 2015-05-04 10:47:31,213 - XmlConfig['hive-site.xml'] {'group': 'hadoop', 'conf_dir':
'/etc/hive/conf.server', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hive', 'configurations':
...}
>>> 2015-05-04 10:47:31,236 - Generating config: /etc/hive/conf.server/hive-site.xml
>>> 2015-05-04 10:47:31,237 - File['/etc/hive/conf.server/hive-site.xml'] {'owner':
'hive', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
>>> 2015-05-04 10:47:31,245 - Writing File['/etc/hive/conf.server/hive-site.xml']
because contents don't match
>>> 2015-05-04 10:47:31,251 - File['/etc/hive/conf.server/hive-env.sh'] {'content':
InlineTemplate(...), 'owner': 'hive', 'group': 'hadoop'}
>>> 2015-05-04 10:47:31,253 - Execute['hive mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/
; rm -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp /usr/share/java/mysql-connector-java.jar
/usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] {'environment': ..., 'path': ['/bin',
'/usr/bin/'], 'creates': '/usr/hdp/current/hive-client/lib/mysql-connector-java.jar', 'not_if':
'test -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar'}
>>> 2015-05-04 10:47:31,282 - Skipping Execute['hive mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/
; rm -f /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ; cp /usr/share/java/mysql-connector-java.jar
/usr/hdp/current/hive-client/lib/mysql-connector-java.jar'] due to not_if
>>> 2015-05-04 10:47:31,285 - Execute['/bin/sh -c 'cd /usr/lib/ambari-agent/ &&
curl -kf -x "" --retry 5 http://sv2lxbdp2mst05.corp.equinix.com:8080/resources/DBConnectionVerification.jar
-o DBConnectionVerification.jar''] {'environment': ..., 'not_if': '[ -f DBConnectionVerification.jar]'}
>>> 2015-05-04 10:47:31,370 - File['/var/lib/ambari-agent/data/tmp/start_hiveserver2_script']
{'content': Template('startHiveserver2.sh.j2'), 'mode': 0755}
>>> 2015-05-04 10:47:31,372 - Directory['/var/run/hive'] {'owner': 'hive', 'group':
'hadoop', 'mode': 0755, 'recursive': True}
>>> 2015-05-04 10:47:31,372 - Directory['/var/log/hive'] {'owner': 'hive', 'group':
'hadoop', 'mode': 0755, 'recursive': True}
>>> 2015-05-04 10:47:31,373 - Directory['/var/lib/hive'] {'owner': 'hive', 'group':
'hadoop', 'mode': 0755, 'recursive': True}
>>> 2015-05-04 10:47:31,453 - Could not verify HDP version by calling '/usr/bin/hdp-select
versions > /tmp/tmpTrvIN3'. Return Code: 0, Output: 2.2.4.2-2
>>> .
>>> 2015-05-04 10:47:31,531 - Could not verify HDP version by calling '/usr/bin/hdp-select
versions > /tmp/tmpHFa97f'. Return Code: 0, Output: 2.2.4.2-2
>>> .
>>> 2015-05-04 10:47:31,600 - Execute['env JAVA_HOME=/usr/jdk64/jdk1.7.0_67 /var/lib/ambari-agent/data/tmp/start_hiveserver2_script
/var/log/hive/hive-server2.out /var/log/hive/hive-server2.log /var/run/hive/hive-server.pid
/etc/hive/conf.server /var/log/hive'] {'environment': ..., 'not_if': 'ls /var/run/hive/hive-server.pid
>/dev/null 2>&1 && ps `cat /var/run/hive/hive-server.pid` >/dev/null
2>&1', 'user': 'hive', 'path': ['/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/usr/hdp/current/hive-client/bin:/usr/hdp/current/hadoop-client/bin']}
>>> 2015-05-04 10:47:31,693 - Execute['/usr/jdk64/jdk1.7.0_67/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar
org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://sv2lxbdp2mst04.corp.equinix.com/hive?createDatabaseIfNotExist=true'
hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'],
'tries': 5, 'try_sleep': 10}
>>>
>>>
>>
>

Mime
View raw message