hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13374) HiveServer2 hangs up if query to scan too many partition is submitted
Date Thu, 27 Oct 2016 21:25:59 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613273#comment-15613273
] 

Thejas M Nair commented on HIVE-13374:
--------------------------------------

[~pxiong] If this is resulting in longer compile phase time, HS2 would block other query compiles
before 2.0 (HIVE-4239) , as there was a lock around compile phase.
Thats likely one part of the issue.

The other part seems to be memory consumption is very high when very large number (10s of
thousands) of partitions are queried.


> HiveServer2 hangs up if query to scan too many partition is submitted
> ---------------------------------------------------------------------
>
>                 Key: HIVE-13374
>                 URL: https://issues.apache.org/jira/browse/HIVE-13374
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 1.2.1
>         Environment: HDP2.4.0
> JDK1.8
>            Reporter: Wataru Yukawa
>
> If query to scan too many partition(for example, more than 40,000 partitions) is submitted,
hiveserver2 hangs up
> hiveserver2 java heap status is the following.
> https://gyazo.com/e0a35bd628a0dc904bc24f19cdb2261e
> https://gyazo.com/65f12cd3875330995508ae11afd394a2
> OutOfMemoryError doesn't occur. but other queries fails.
> Maybe it is necessary to spill to disk...
> hiveserver2.log
> {code}
> ....
> 2016-03-28 13:44:28,923 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getSplits(517)) - Total number of paths: 47409, launching 50
threads to check non-combinable ones.
> 2016-03-28 13:44:29,444 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getCombineSplits(439)) - CombineHiveInputSplit creating pool
for hdfs://.../aaa=.../yyyymmdd=20150721; using filter path hdfs://.../aaa=.../yyyymmdd=20150721
> ....
> 2016-03-28 13:44:34,811 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already
created for hdfs://.../aaa=.../yyyymmdd=20160326; using filter path hdfs://.../aaa=.../yyyymmdd=20160326
> 2016-03-28 13:44:34,811 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already
created for hdfs://.../aaa=.../yyyymmdd=20160327; using filter path hdfs://.../aaa=.../yyyymmdd=20160327
> 2016-03-28 13:44:34,811 INFO  [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat
(CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already
created for hdfs://.../aaa=.../yyyymmdd=20160328; using filter path hdfs://.../aaa=.../yyyymmdd=20160328
> 2016-03-28 13:45:00,285 INFO  [org.apache.hadoop.util.JvmPauseMonitor$Monitor@6f5d0190]:
util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine
(eg GC): pause of approximately 4587ms
> GC pool 'PS MarkSweep' had collection(s): count=1 time=4699ms
> GC pool 'PS Scavenge' had collection(s): count=1 time=63ms
> ....
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message