ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hurley" <jhur...@hortonworks.com>
Subject Review Request 41152: HiveServer Upgrade Fails Because Of Invalid Token Store Class
Date Wed, 09 Dec 2015 22:16:52 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41152/
-----------------------------------------------------------

Review request for Ambari, Alejandro Fernandez, Nate Cole, and Sumit Mohanty.


Bugs: AMBARI-14303
    https://issues.apache.org/jira/browse/AMBARI-14303


Repository: ambari


Description
-------

*Steps*
Setup HDP 2.1 cluster with Ambari 2.2.0; Enable security on the cluster
Modify the relevant DB tables
Start Express Upgrade to 2.3.4 and let it complete
Observe the status of HIve service components

*Result:*
All Hive servers report down
When I tried to start them they failed to come up (see attached hive log)

This one is is a permission error in a kerberized cluster.

Configs,
hive.server2.thrift.http.port : 10001
hive.server2.thrift.port : 10000
hive.cluster.delegation.token.store.class : org.apache.hadoop.hive.thrift.ZooKeeperTokenStore

That last config should instead be org.apache.hadoop.hive.thrift.DBTokenStore in a kerberized
cluster.
After EU/RU, the config datanucleus.rdbms.datastoreAdapterClassName did not have a value.

Nothing was using those ports,
[root@os-r6-oncjcs-baikaltom20sechanr-1 ~]# netstat -anp | grep 10000
[root@os-r6-oncjcs-baikaltom20sechanr-1 ~]# netstat -anp | grep 10001

I then restarted HiveServer2, which reported success in Ambari, but the process then died.

Hive Server 2 log,
{code}
2015-12-09 18:41:21,321 FATAL [Thread-10]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101))
- Error starting HiveServer2: could not start ThriftBinaryCLIService
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path
/hive/cluster/delegationHIVESERVER2/keys
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.startDelegationTokenSecretManager(HadoopThriftAuthBridge.java:444)
        at org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:124)
        at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:57)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed
for /hive/cluster/delegationHIVESERVER2/keys
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:672)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
        ... 6 more
2015-12-09 18:41:21,323 FATAL [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(203))
- Unable to create HiveServer2 namespace: hiveserver2 on ZooKeeper
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for
/hiveserver2
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:672)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
        at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:198)
        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:417)
        at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:78)
        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:654)
        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:527)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{code}


Diffs
-----

  ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml 2341dcc

  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/config-upgrade.xml 8c3e7da 

Diff: https://reviews.apache.org/r/41152/diff/


Testing
-------

Upgraded HDP 2.1 with Hive and verified that the new value is correctly set.


Thanks,

Jonathan Hurley


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message