hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Federico D'Ambrosio" <fedex...@gmail.com>
Subject Non-local session path expected to be non-null using storm-hive
Date Mon, 10 Jul 2017 15:14:36 GMT
Greetings,
I'm trying to get a working dataflow stack on a 6 node cluster (2 masters +
4 slaves, no Kerberos) using Kafka (2.10_0.10), Storm (1.0.1) and Hive2
(1.2.1). Storm is able to communicate with Kafka, but can't seemingly
operate on Hive (on master-1), even though it manages to connect to its
metastore.

I thought originally it was a problem of permissions on either HDFS or the
local filesystem, but even though I set 777 permissions on /tmp/hive,
there's still this issue.

In core-site.xml:

   - hadoop.proxyuser.hcat.group
   -

   hadoop.proxyuser.hcat.hosts
   -

   hadoop.proxyuser.hdfs.groups
   - hadoop.proxyuser.hdfs.hosts
   - hadoop.proxyuser.hive.groups
   - hadoop.proxyuser.hive.hosts
   - hadoop.proxyuser.root.groups
   - hadoop.proxyuser.root.hosts

are all set to '*'.

Hive2, as far as I see is correctly set to work with transactions, being
the target table with transactional=true, stored as orc and bucketed. In
the hive-site.xml:

   - hive.compactor.worker.threads = 1
   - hive.compactor.initiator.on = true
   - hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager

I get a Nullpointer Exception, you may find the stack trace among the
attached files.

>From what I can gather, the NullpointerException is thrown in the following
method inside SessionState:


   1. public static Path getHDFSSessionPath(Configuration conf) {
   2. 	SessionState ss = SessionState.get();
   3. 	if (ss == null) {
   4. 		String sessionPathString = conf.get(HDFS_SESSION_PATH_KEY);
   5. 		Preconditions.checkNotNull(sessionPathString,  "Conf non-local
session path expected to be non-null");
   6. 		return new Path(sessionPathString);
   7. 	}
   8. 	Preconditions.checkNotNull(ss.hdfsSessionPath,  "Non-local
session path expected to be non-null");
   9. 	return ss.hdfsSessionPath;
   10. }


Specifically, by:


   1. Preconditions.checkNotNull(ss.hdfsSessionPath, "Non-local
session path expected to be non-null");

So, it seems to be an hdfs related issue, but I can't understand why it's
happening.

>From what I gather, this occurs when Hive tries to retrieve the local path
of the session, which is stored in the _hive.local.session.path
configuration variable. The value of this variable is assigned each time a
new Hive session is created, and it is formed by merging the path for user
temporary files (hive.exec.local.scratchdir) to the session ID (
hive.session.id).

If indeed is a permissions issue, what should I look into to find the
origin of the issue?

Thanks for your help,
Federico

Mime
View raw message