drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Altekruse (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3306) Concurrently Running hive queries results in a deadlock situation
Date Wed, 17 Jun 2015 23:23:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590834#comment-14590834
] 

Jason Altekruse commented on DRILL-3306:
----------------------------------------

[~rkins] Please remember to set the fix version, we should be checking priority of all bugs,
but as we get close to a release we might be filtering on issues just tagged for this release.
We definitely should be looking at this before 1.1 is wrapped up.

> Concurrently Running hive queries results in a deadlock situation
> -----------------------------------------------------------------
>
>                 Key: DRILL-3306
>                 URL: https://issues.apache.org/jira/browse/DRILL-3306
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - Hive
>            Reporter: Rahul Challapalli
>            Assignee: Venki Korukanti
>            Priority: Critical
>             Fix For: 1.1.0
>
>         Attachments: error.log
>
>
> I tried submitting ~20 queries using 10 concurrent threads. Now the drillbit goes into
an unresponsive state. Now even trying to launch a new sqlline session hangs (until hive.metastore.client.socket.timeout
expires)
> Below is the jstack output
> {code}
> Attaching to process ID 4662, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 24.65-b04
> Deadlock Detection:
> java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x00007fc0f8630000
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 15 more
> Can't print deadlocks:Unable to deduce type of thread from address 0x00007fc0f8630000
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	... 6 more
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x00007fc0f8630000
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 12 more
> {code}
> Jstack Output of the hive metastore
> {code}
> Attaching to process ID 32320, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 24.65-b04
> Deadlock Detection:
> java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x00007f1abd191800
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 15 more
> Can't print deadlocks:Unable to deduce type of thread from address 0x00007f1abd191800
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	... 6 more
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x00007f1abd191800
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 12 more
> {code}
> Observing from the web ui, after every 10 mins one query changes state from pending to
running. All the queries in the running state did not complete and I stopped the drillbit
after waiting for 30 mins
> Also at this stage attempting to stop the drillbit does not succeed and the script internally
call the kill command on the process
> I attached the contents of drillbit.log file. Let me know if you need anything else



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message