hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wataru Yukawa (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-13374) HiveServer2 hangs up if query to scan too many partition is submitted
Date Tue, 29 Mar 2016 01:51:25 GMT
Wataru Yukawa created HIVE-13374:
------------------------------------

             Summary: HiveServer2 hangs up if query to scan too many partition is submitted
                 Key: HIVE-13374
                 URL: https://issues.apache.org/jira/browse/HIVE-13374
             Project: Hive
          Issue Type: Bug
          Components: HiveServer2
    Affects Versions: 1.2.1
         Environment: HDP2.4.0
JDK1.8
            Reporter: Wataru Yukawa
            Assignee: Vaibhav Gumashta


If query to scan too many partition(for example, more than 40,000 partitions) is submitted,
hiveserver2 hangs up

hiveserver2 java heap status is the following.
https://gyazo.com/e0a35bd628a0dc904bc24f19cdb2261e
https://gyazo.com/65f12cd3875330995508ae11afd394a2

OutOfMemoryError doesn't occur. but other queries fails.

Maybe it is necessary to spill to disk...

hiveserver2.log
{code}
....
2016-03-28 13:44:28,923 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getSplits(517)) - Total number of paths: 47409, launching 50
threads to check non-combinable ones.
2016-03-28 13:44:29,444 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getCombineSplits(439)) - CombineHiveInputSplit creating pool
for hdfs://.../aaa=.../yyyymmdd=20150721; using filter path hdfs://.../aaa=.../yyyymmdd=20150721
....
2016-03-28 13:44:34,811 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already
created for hdfs://.../aaa=.../yyyymmdd=20160326; using filter path hdfs://.../aaa=.../yyyymmdd=20160326
2016-03-28 13:44:34,811 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already
created for hdfs://.../aaa=.../yyyymmdd=20160327; using filter path hdfs://.../aaa=.../yyyymmdd=20160327
2016-03-28 13:44:34,811 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already
created for hdfs://.../aaa=.../yyyymmdd=20160328; using filter path hdfs://.../aaa=.../yyyymmdd=20160328
2016-03-28 13:45:00,285 INFO  [org.apache.hadoop.util.JvmPauseMonitor$Monitor@6f5d0190]: util.JvmPauseMonitor
(JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of
approximately 4587ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=4699ms
GC pool 'PS Scavenge' had collection(s): count=1 time=63ms
....
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message