hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (Jira)" <>
Subject [jira] [Commented] (HIVE-23164) server is not properly terminated because of non-daemon threads
Date Wed, 22 Apr 2020 13:02:00 GMT


Hive QA commented on HIVE-23164:

Here are the results of testing the latest attachment:

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17133 tests executed
*Failed tests:*
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2] (batchId=37)

Test results:
Console output:
Test logs:

Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed

This message is automatically generated.

ATTACHMENT ID: 13000727 - PreCommit-HIVE-Build

> server is not properly terminated because of non-daemon threads
> ---------------------------------------------------------------
>                 Key: HIVE-23164
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>            Reporter: Eugene Chung
>            Assignee: Eugene Chung
>            Priority: Major
>         Attachments: HIVE-23164.01.patch, HIVE-23164.02.patch, HIVE-23164.03.patch, thread_dump_hiveserver2_is_not_terminated.txt
> HiveServer2 which receives the deregister command is at first preparing shutdown. If
there's no remaining session, HiveServer2.stop() is called to shut down. But I found the case
that the HiveServer2 JVM is not terminated even if HiveServer2.stop() has been called and
processed. The case is always occurred when the local(embedded) metastore is used.
> I've attached the full thread dump describing the situation.
> [^thread_dump_hiveserver2_is_not_terminated.txt]
> In this thread dump, you can see some bunch of 'daemon' threads, NO main thread, and
some 'non-daemon' thread(or user thread)s. As specified by [],
if there is at least one user thread exists, JVM does not terminate. (Note that DestroyJavaVM
thread is non-daemon but it's special.)
> {code:java}
> "pool-8-thread-1" #24 prio=5 os_prio=0 tid=0x00007f52ad1fc000 nid=0x821c waiting on condition
>  java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for <0x00000003cfa057c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(
>  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(
>  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
>  at java.util.concurrent.ThreadPoolExecutor.getTask(
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(
>  at java.util.concurrent.ThreadPoolExecutor$
>  at
> Locked ownable synchronizers:
>  - None
> {code}
> The thread above is created by the ScheduledThreadPoolExecutor(int coreSize) constructor
with default ThreadFactory which always makes a thread non-daemon. If such thread pool is
not destroyed by calling ScheduledThreadPoolExecutor.shutdown() method, JVM cannot terminate!
The only way to kill is TERM signal. If JVM receives TERM signal, it ignores non-daemon threads
and terminates.
> So I have been digging modules which create ScheduledThreadPoolExecutor with non-daemon
threads and I got it. As you may guess, it's the local(embedded) metastore. ScheduledThreadPoolExecutor
is created by org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler#startAlwaysTaskThreads()
and ScheduledThreadPoolExecutor.shutdown() is never called.
> Plus, I found another usage of creating such ScheduledThreadPoolExecutor and not calling
its shutdown. 

This message was sent by Atlassian Jira

View raw message