drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venki Korukanti (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-3306) Concurrently Running hive queries results in a deadlock situation
Date Sun, 21 Jun 2015 07:16:00 GMT

     [ https://issues.apache.org/jira/browse/DRILL-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Venki Korukanti updated DRILL-3306:
-----------------------------------
    Attachment: DRILL-3306-1.patch

We use single metastore client connection across the threads and more than one thread can
make an RPC call using the single connection. Problem is thrift client (metastore client)
is not thread safe. More on this [here|http://comments.gmane.org/gmane.comp.lib.thrift.user/2704].
Fix is to synchronize to have only one RPC call at anytime.

> Concurrently Running hive queries results in a deadlock situation
> -----------------------------------------------------------------
>
>                 Key: DRILL-3306
>                 URL: https://issues.apache.org/jira/browse/DRILL-3306
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - Hive
>            Reporter: Rahul Challapalli
>            Assignee: Venki Korukanti
>            Priority: Critical
>             Fix For: 1.1.0
>
>         Attachments: DRILL-3306-1.patch, error.log
>
>
> I tried submitting ~20 queries using 10 concurrent threads. Now the drillbit goes into
an unresponsive state. Now even trying to launch a new sqlline session hangs (until hive.metastore.client.socket.timeout
expires)
> Below is the jstack output
> {code}
> Attaching to process ID 4662, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 24.65-b04
> Deadlock Detection:
> java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x00007fc0f8630000
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 15 more
> Can't print deadlocks:Unable to deduce type of thread from address 0x00007fc0f8630000
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	... 6 more
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x00007fc0f8630000
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 12 more
> {code}
> Jstack Output of the hive metastore
> {code}
> Attaching to process ID 32320, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 24.65-b04
> Deadlock Detection:
> java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x00007f1abd191800
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 15 more
> Can't print deadlocks:Unable to deduce type of thread from address 0x00007f1abd191800
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	... 6 more
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x00007f1abd191800
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 12 more
> {code}
> Observing from the web ui, after every 10 mins one query changes state from pending to
running. All the queries in the running state did not complete and I stopped the drillbit
after waiting for 30 mins
> Also at this stage attempting to stop the drillbit does not succeed and the script internally
call the kill command on the process
> I attached the contents of drillbit.log file. Let me know if you need anything else



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message