drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Deneche A. Hakim (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3842) JVM dies if drill attempts to read too many files in the directory that blow up heap
Date Sun, 10 Apr 2016 05:10:25 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233883#comment-15233883
] 

Deneche A. Hakim commented on DRILL-3842:
-----------------------------------------

Looking through the heap dump, most of the usage seems to be related to the parquet metadata
for all files. After forcing a full GC using an external tool, heap usage went down to about
60MB which proves we are not leaking any memory, we are just using too much when gathering
the parquet metadata.

I generated the metadata cache, after that the query finishes in less than 10 seconds.

> JVM dies if drill attempts to read too many files in the directory that blow up heap

> -------------------------------------------------------------------------------------
>
>                 Key: DRILL-3842
>                 URL: https://issues.apache.org/jira/browse/DRILL-3842
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Flow
>    Affects Versions: 1.1.0, 1.2.0
>            Reporter: Victoria Markman
>            Assignee: Deneche A. Hakim
>            Priority: Critical
>
> Run {{select count(*) from t1}} where  t1 directory consists of 1.9 million little parquet
files. The outcome: drillbit is dead and out of working set.
> 1. Client never got response back from the server
> 2. drillbit.log
> {code}
> 2015-09-25 17:56:55,935 [29fa756f-894d-0340-3661-b925bff0fe11:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Took 47999 ms to get file statuses
> 2015-09-25 18:43:19,871 [BitServer-4] INFO  o.a.d.exec.rpc.control.ControlServer - RPC
connection /10.10.88.135:31011 <--> /10.10.88.135:51675 (control server) timed out.
 Timeout was set to 300 seconds. Closing connection.
> 2015-09-25 18:50:06,026 [BitServer-3] INFO  o.a.d.exec.rpc.control.ControlClient - Channel
closed /10.10.88.135:51675 <--> /10.10.88.135:31011.
> 2015-09-25 18:50:06,032 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception
in RPC communication.  Connection: /10.10.88.135:31010 <--> /10.10.88.133:51612 (user
client).  Closing connection.
> java.lang.OutOfMemoryError: Java heap space
> {code}
> drillbit.out
> {code}
> Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread
"main-SendThread(atsqa4-133.qa.lab:5181)"
> Exception in thread "WorkManager.StatusThread" java.lang.OutOfMemoryError: Java heap
space
> 2015-09-25 18:53:52
> Full thread dump OpenJDK 64-Bit Server VM (24.65-b04 mixed mode):
> {code}
> jstack
> {code}
> [Fri Sep 25 18:53:29 ] # jstack 63205 
> 63205: Unable to open socket file: target process not responding or HotSpot VM not loaded
> The -F option can be used when the target process is not responding
> {code}
> jstack -F
> {code}
> Attaching to process ID 63205, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 24.65-b04
> java.lang.RuntimeException: Unable to deduce type of thread from address 0x0000000004093800
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
> 	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x0000000004093800
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 15 more
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> 	at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x0000000004093800
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> 	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
> 	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> 	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> 	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> 	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> 	... 6 more
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x0000000004093800
> 	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> 	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> 	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> 	... 12 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message