hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zengyongping (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-8440) Hiveserver block query
Date Mon, 13 Oct 2014 09:11:33 GMT

     [ https://issues.apache.org/jira/browse/HIVE-8440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

zengyongping updated HIVE-8440:
-------------------------------
    Description: 
In our  environment Hiveserver handle approximately 15000 jobs per day。Sometimes we found
a large number of  ETL tasks wait in our ETL schedule  queue to run。In this situation only
few Hadoop Job running in Hadoop cluster,many maps and reduces slots free.We view Hiveserver
heap found Hiveserver block a large number of query:
"pool-1-thread-17903" prio=10 tid=0x00007f89589c8000 nid=0x13865 waiting for monitor entry
[0x00007f893413b000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
	- waiting to lock <0x00000006000ac458> (a java.lang.Object)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
	at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
	at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
	at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

     It is a problem of HiveServer cannot handle concurrent requests from more than one client.?Why
HiveServer cannot handle concurrent requests?
https://github.com/apache/hive/blob/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L975
Under code in org.apache.hadoop.hive.ql cause Hiveserver cannot handle concurrent requests?
 int ret;
    synchronized (compileMonitor) {
      ret = compile(command);
    }



  was:
In our  environment Hiveserver handle approximately 15000 jobs per day。Sometimes we found
a large number of  ETL tasks wait in our ETL schedule  queue to run。In this situation only
few Hadoop Job in Hadoop cluster,many maps and reduces slots free.we view Hiveserver heap
found Hiveserver block a large number of query:
"pool-1-thread-17903" prio=10 tid=0x00007f89589c8000 nid=0x13865 waiting for monitor entry
[0x00007f893413b000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
	- waiting to lock <0x00000006000ac458> (a java.lang.Object)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
	at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
	at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
	at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

     It is a problem of HiveServer cannot handle concurrent requests from more than one client.?Why
HiveServer cannot handle concurrent requests?
https://github.com/apache/hive/blob/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L975
Under code in org.apache.hadoop.hive.ql cause Hiveserver cannot handle concurrent requests?
 int ret;
    synchronized (compileMonitor) {
      ret = compile(command);
    }



    Environment: 
OS:Red Hat Enterprise Linux Server release 6.5 (Santiago)
CPU:64 cores
Memory:64GB
Hadoop:CDH4.2.1

  was:
OS:Red Hat Enterprise Linux Server release 6.5 (Santiago)
CPU:64 cores
Memory:64GB



> Hiveserver block query
> ----------------------
>
>                 Key: HIVE-8440
>                 URL: https://issues.apache.org/jira/browse/HIVE-8440
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.12.0
>         Environment: OS:Red Hat Enterprise Linux Server release 6.5 (Santiago)
> CPU:64 cores
> Memory:64GB
> Hadoop:CDH4.2.1
>            Reporter: zengyongping
>            Priority: Critical
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> In our  environment Hiveserver handle approximately 15000 jobs per day。Sometimes we
found a large number of  ETL tasks wait in our ETL schedule  queue to run。In this situation
only few Hadoop Job running in Hadoop cluster,many maps and reduces slots free.We view Hiveserver
heap found Hiveserver block a large number of query:
> "pool-1-thread-17903" prio=10 tid=0x00007f89589c8000 nid=0x13865 waiting for monitor
entry [0x00007f893413b000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
> 	- waiting to lock <0x00000006000ac458> (a java.lang.Object)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
> 	at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
> 	at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
> 	at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
>      It is a problem of HiveServer cannot handle concurrent requests from more than one
client.?Why HiveServer cannot handle concurrent requests?
> https://github.com/apache/hive/blob/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L975
> Under code in org.apache.hadoop.hive.ql cause Hiveserver cannot handle concurrent requests?
>  int ret;
>     synchronized (compileMonitor) {
>       ret = compile(command);
>     }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message