hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thai Bui (JIRA)" <>
Subject [jira] [Created] (HIVE-17501) Hive should allow default Tez sessions to be skipped
Date Mon, 11 Sep 2017 14:18:00 GMT
Thai Bui created HIVE-17501:

             Summary: Hive should allow default Tez sessions to be skipped
                 Key: HIVE-17501
             Project: Hive
          Issue Type: Bug
          Components: Tez
    Affects Versions: 2.2.0, 2.1.1
         Environment: HDP, Hue 4.0
            Reporter: Thai Bui

Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be skipped mostly
because of this line

However, some clients such as Hue 4, allow multiple sessions to be used per user. Under this
configuration, a Thrift client will send a request to either reuse or open a new session.
The reuse request could include the session id of a currently used snippet being executed
in Hue, this causes HS2 to throw an exception:

2017-09-10T17:51:36,548 INFO  [Thread-89]: tez.TezSessionPoolManager (
- The current user: hive, session user: hive
2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task ( - Failed
to execute tez graph.
org.apache.hadoop.hive.ql.metadata.HiveException: The pool session sessionId=5b61a578-6336-41c5-860d-9838166f97fe,
queueName=llap, user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591015330ms
should have been returned to the pool
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(
	at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute( [hive-exec-]
	at org.apache.hadoop.hive.ql.exec.Task.executeTask( [hive-exec-]
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential( [hive-exec-]
	at [hive-exec-]

Note that every query is issued as a single 'hive' user to share the LLAP daemon pool, a set
of pre-determined number of AMs is initialized at setup time. Thus, HS2 should allow new sessions
from a Thrift client to be used out of the pool, or an existing session to be skipped and
an unused session from the pool to be returned. The logic to throw an exception in the  `canWorkWithSameSession`
doesn't make sense to me.

I have a solution to fix this issue in my local branch at
When applied, the log will become like so

2017-09-10T09:15:33,578 INFO  [Thread-239]: tez.TezSessionPoolManager (
- Skipping default session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms since it is being

A test case is provided in my branch to demonstrate how it works. If possible I would like
this patch to be applied to version 2.1, 2.2 and master. Since we are using 2.1 LLAP in production
with Hue 4, this patch is critical to our success.

Alternatively, if this patch is too broad in scope, I propose adding an option to allow "skipping
of currently used default sessions". With this new option default to "false", existing behavior
won't change unless the option is turned on.

P/s: I'm not an contributor &/ committer, this will be my first time contributing to Hive
and the Apache foundation. Any early review is greatly appreciated, thanks!

This message was sent by Atlassian JIRA

View raw message