atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayub Pathan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ATLAS-2199) DeadLock in atlas while producing metadata with hive view > 300 cols.
Date Tue, 10 Oct 2017 10:04:00 GMT
Ayub Pathan created ATLAS-2199:
----------------------------------

             Summary: DeadLock in atlas while producing metadata with hive view > 300 cols.
                 Key: ATLAS-2199
                 URL: https://issues.apache.org/jira/browse/ATLAS-2199
             Project: Atlas
          Issue Type: Bug
          Components:  atlas-core
    Affects Versions: 0.8.2
            Reporter: Ayub Pathan
            Priority: Critical
             Fix For: 0.8.2
         Attachments: application.log, atlas.20171009-192118.err, atlas.20171009-192118.out,
thread-dump.txt

Steps to reproduce:
1. Produce messages using hive hook - create view with 300 columns. Example, command below
{noformat}
create table t1_300(col0 string, col1 string, col2 string, col3 string, col4 string, col5
string, col6 string, col7 string, col8 string, col9 string, col10 string, col11 string, col12
string, col13 string, col14 string, col15 string, col16 string, col17 string, col18 string,
col19 string, col20 string, col21 string, col22 string, col23 string, col24 string, col25
string, col26 string, col27 string, col28 string, col29 string, col30 string, col31 string,
col32 string, col33 string, col34 string, col35 string, col36 string, col37 string, col38
string, col39 string, col40 string, col41 string, col42 string, col43 string, col44 string,
col45 string, col46 string, col47 string, col48 string, col49 string, col50 string, col51
string, col52 string, col53 string, col54 string, col55 string, col56 string, col57 string,
col58 string, col59 string, col60 string, col61 string, col62 string, col63 string, col64
string, col65 string, col66 string, col67 string, col68 string, col69 string, col70 string,
col71 string, col72 string, col73 string, col74 string, col75 string, col76 string, col77
string, col78 string, col79 string, col80 string, col81 string, col82 string, col83 string,
col84 string, col85 string, col86 string, col87 string, col88 string, col89 string, col90
string, col91 string, col92 string, col93 string, col94 string, col95 string, col96 string,
col97 string, col98 string, col99 string, col100 string, col101 string, col102 string, col103
string, col104 string, col105 string, col106 string, col107 string, col108 string, col109
string, col110 string, col111 string, col112 string, col113 string, col114 string, col115
string, col116 string, col117 string, col118 string, col119 string, col120 string, col121
string, col122 string, col123 string, col124 string, col125 string, col126 string, col127
string, col128 string, col129 string, col130 string, col131 string, col132 string, col133
string, col134 string, col135 string, col136 string, col137 string, col138 string, col139
string, col140 string, col141 string, col142 string, col143 string, col144 string, col145
string, col146 string, col147 string, col148 string, col149 string, col150 string, col151
string, col152 string, col153 string, col154 string, col155 string, col156 string, col157
string, col158 string, col159 string, col160 string, col161 string, col162 string, col163
string, col164 string, col165 string, col166 string, col167 string, col168 string, col169
string, col170 string, col171 string, col172 string, col173 string, col174 string, col175
string, col176 string, col177 string, col178 string, col179 string, col180 string, col181
string, col182 string, col183 string, col184 string, col185 string, col186 string, col187
string, col188 string, col189 string, col190 string, col191 string, col192 string, col193
string, col194 string, col195 string, col196 string, col197 string, col198 string, col199
string, col200 string, col201 string, col202 string, col203 string, col204 string, col205
string, col206 string, col207 string, col208 string, col209 string, col210 string, col211
string, col212 string, col213 string, col214 string, col215 string, col216 string, col217
string, col218 string, col219 string, col220 string, col221 string, col222 string, col223
string, col224 string, col225 string, col226 string, col227 string, col228 string, col229
string, col230 string, col231 string, col232 string, col233 string, col234 string, col235
string, col236 string, col237 string, col238 string, col239 string, col240 string, col241
string, col242 string, col243 string, col244 string, col245 string, col246 string, col247
string, col248 string, col249 string, col250 string, col251 string, col252 string, col253
string, col254 string, col255 string, col256 string, col257 string, col258 string, col259
string, col260 string, col261 string, col262 string, col263 string, col264 string, col265
string, col266 string, col267 string, col268 string, col269 string, col270 string, col271
string, col272 string, col273 string, col274 string, col275 string, col276 string, col277
string, col278 string, col279 string, col280 string, col281 string, col282 string, col283
string, col284 string, col285 string, col286 string, col287 string, col288 string, col289
string, col290 string, col291 string, col292 string, col293 string, col294 string, col295
string, col296 string, col297 string, col298 string, col299 string);
create view v_t1_300 as select * from t1_300;
{noformat}

2. With this Atlas throws out of memory exception. Below error log confirming the same.
{noformat}
[root@ctr-e134-1499953498516-209460-01-000004 atlas]# cat atlas.20171009-192118.err
Exception in thread "pool-2-thread-6" Exception in thread "pool-2-thread-2" java.lang.OutOfMemoryError:
Java heap space
	at java.util.HashMap$KeySet.iterator(HashMap.java:916)
	at java.util.HashSet.iterator(HashSet.java:172)
	at sun.nio.ch.Util$3.iterator(Util.java:324)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-22" Exception in thread "pool-2-thread-3" Exception in thread
"HashSessionScavenger-0" java.lang.OutOfMemoryError: Java heap space
	at java.util.HashMap$KeySet.iterator(HashMap.java:916)
	at java.util.HashSet.iterator(HashSet.java:172)
	at sun.nio.ch.Util$3.iterator(Util.java:324)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: Java heap space
	at java.util.concurrent.ConcurrentHashMap$ValuesView.iterator(ConcurrentHashMap.java:4683)
	at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:314)
	at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:285)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
java.lang.OutOfMemoryError: Java heap space
	at java.util.HashMap$KeySet.iterator(HashMap.java:916)
	at java.util.HashSet.iterator(HashSet.java:172)
	at sun.nio.ch.Util$3.iterator(Util.java:324)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-2-thread-7" java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-2-thread-1" java.lang.OutOfMemoryError: Java heap space
[root@ctr-e134-1499953498516-209460-01-000004 atlas]#
{noformat}

3. After this step, Atlas became unresponsive, no requests are honoured. All requests are
timedout.	
4. JSTACK output confirming the deadlock.
{noformat}
[root@ctr-e134-1499953498516-209460-01-000004 atlas]# ps axu | grep Atlas
atlas     131887  3.6  0.9 20609684 2612572 ?    Sl   Oct09  31:03 /usr/lib/jvm/java-openjdk/bin/java
-Datlas.log.dir=/grid/0/log/atlas -Datlas.log.file=application.log -Datlas.home=/usr/hdp/.3.0-175/atlas
-Datlas.conf=/usr/hdp/current/atlas-server/conf -Xms2048m -Xmx2048m -XX:MaxNewSize=600m -XX:MetaspaceSize=100m
-XX:MaxMetaspaceSize=512m -server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/grid/0/log/atlas/atlas_server.hprof -Xloggc:/grid/0/log/atlas/gc-worker.log
-verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails
-XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -Dlog4j.configuration=atlas-log4j.xml -Djava..auth.login.config=/usr/hdp/current/atlas-server/conf/atlas_jaas.conf
-classpath /usr/hdp/current/atlas-server/conf:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/classes:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib/*:/usr/hdp/.3.0-175/atlas/libext/*:/etc/hbase/conf
org.apache.atlas.Atlas -app /usr/hdp/current/atlas-server/server/webapp/atlas
root     1666487  0.0  0.0  10632   968 pts/1    S+   09:34   0:00 grep --color=auto Atlas
[root@ctr-e134-1499953498516-209460-01-000004 atlas]#
{noformat}

{noformat}
[root@ctr-e134-1499953498516-209460-01-000004 atlas]# /usr/lib/jvm/java-openjdk/bin/jstack
-F -l 131887
Attaching to process ID 131887, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.144-b01
Deadlock Detection:

java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f8838013800
(expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:169)
	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:153)
	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:62)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
	at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
	at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
	at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
	at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address
0x00007f8838013800
	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:165)
	... 17 more
Can't print deadlocks:Unable to deduce type of thread from address 0x00007f8838013800 (expected
type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
	at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: java.lang.InternalError: Metadata does not appear to be polymorphic
	at sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:278)
	at sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102)
	at sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:68)
	at sun.jvm.hotspot.memory.DictionaryEntry.klass(DictionaryEntry.java:71)
	at sun.jvm.hotspot.memory.DictionaryEntry.equals(DictionaryEntry.java:79)
	at sun.jvm.hotspot.memory.Dictionary.getEntry(Dictionary.java:99)
	at sun.jvm.hotspot.memory.Dictionary.find(Dictionary.java:87)
	at sun.jvm.hotspot.memory.SystemDictionary.find(SystemDictionary.java:159)
	at sun.jvm.hotspot.memory.SystemDictionary.find(SystemDictionary.java:151)
	at sun.jvm.hotspot.memory.SystemDictionary.getAbstractOwnableSynchronizerKlass(SystemDictionary.java:135)
	at sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.fillLocks(ConcurrentLocksPrinter.java:66)
	at sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.<init>(ConcurrentLocksPrinter.java:36)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:71)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
	at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
	at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
	at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
	... 6 more
[root@ctr-e134-1499953498516-209460-01-000004 atlas]#
{noformat}

Attaching the thread dump output as well.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message