hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "BELUGA BEHR (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
Date Thu, 16 Aug 2018 20:36:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583034#comment-16583034
] 

BELUGA BEHR commented on HIVE-20401:
------------------------------------

May also be related to "small files"... if the Hive data set has many small files in it, the
Hive compile stage can take a long time to download all of the file information.

> HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
> --------------------------------------------------------------------------------------------
>
>                 Key: HIVE-20401
>                 URL: https://issues.apache.org/jira/browse/HIVE-20401
>             Project: Hive
>          Issue Type: Bug
>          Components: Beeline, HiveServer2
>    Affects Versions: 1.2.2
>            Reporter: zhangbutao
>            Priority: Major
>         Attachments: HiveServer2_jstack.txt
>
>
> HiveServer2  process often gets stuck , and jstack shows about one hundred thread is
blocked at the following code,waiting the resource *0x00000003c0acca40*
> {code:java}
> "HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0 tid=0x00007fd71470c000
nid=0x3e1f waiting for monitor entry [0x00007fd6d6eee000]
> 	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
> 	- waiting to lock <0x00000003c0acca40> (a java.lang.Object)
> 	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
> 	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> 	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
> 	at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> 	at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
> 	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
> 	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> {code}
>  We find one thread locked the resource *0x00000003c0acca40*: ,and the jstack is as follow:
> {code:java}
> "HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0 tid=0x00007fd71433e000
nid=0x37c0 in Object.wait() [0x00007fd6d04b8000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:502)
> 	at org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
> 	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> 	- locked <0x00000003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1351)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> 	at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
> 	at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> 	- locked <0x00000003d9ab5dc8> (a org.apache.hadoop.io.retry.RetryInvocationHandler$Call)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
> 	at com.sun.proxy.$Proxy19.getEZForPath(Unknown Source)
> 	at org.apache.hadoop.hdfs.DFSClient.getEZForPath(DFSClient.java:2810)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$51.doCall(DistributedFileSystem.java:2270)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$51.doCall(DistributedFileSystem.java:2267)
> 	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.getEZForPath(DistributedFileSystem.java:2286)
> 	at org.apache.hadoop.hdfs.client.HdfsAdmin.getEncryptionZoneForPath(HdfsAdmin.java:350)
> 	at org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.isPathEncrypted(Hadoop23Shims.java:1221)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:1883)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:1964)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStagingDirectoryPathname(SemanticAnalyzer.java:1996)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1809)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1544)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10076)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10127)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:210)
> 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:425)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:309)
> 	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1145)
> 	- locked <0x00000003c0acca40> (a java.lang.Object)
> 	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
> 	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> 	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
> 	at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> 	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> 	at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
> 	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
> 	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}
> It seems that Thread-809399 which locked the resource *0x00000003c0acca40* is waiting
for rpc responseļ¼Œ leading to other threads are blocked. Does anyone know what caused this?
  Hadoop version is 2.8.2.  Attachment is full jstack.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message