ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolai Tikhonov <ntikho...@apache.org>
Subject Re: cluster hanged when client node put data into the caches
Date Fri, 08 Dec 2017 11:22:49 GMT
Hello,

On the first look seems that server nodes are fine. Could take a thread
dump via JDK tool (jstack)? Also possible reason it's problem with gc. Can
you enable and share gc logs [1] here?


1. https://apacheignite.readme.io/docs/jvm-and-system-tuning


On Fri, Dec 8, 2017 at 6:24 AM, Aurora <2565003241@qq.com> wrote:

> thread dump:
> ------------------------------------------------------------
> ----------------------------------------------------
>
> Attaching to process ID 12837, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 25.65-b01
> Deadlock Detection:
>
> No deadlocks found.
>
> ----------------- 12838 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5015754      * sun.misc.Unsafe.park(boolean, long) bci:0
> (Interpreted
> frame)
> 0x00007fe3e500798d      *
> java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14
> line:175 (Interpreted frame)
> 0x00007fe3e500798d      *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt()
> bci:1 line:836 (Interpreted frame)
> 0x00007fe3e5007160      *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(int)
> bci:72 line:997 (Interpreted frame)
> 0x00007fe3e500798d      *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(int)
> bci:24 line:1304 (Interpreted frame)
> 0x00007fe3e500798d      * java.util.concurrent.CountDownLatch.await()
> bci:5
> line:231 (Interpreted frame)
> 0x00007fe3e500798d      *
> org.apache.ignite.startup.cmdline.CommandLineStartup.
> main(java.lang.String[])
> bci:309 line:335 (Interpreted frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f9923182
> _ZL17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_
> jmethodIDP18JNI_ArgumentPusherP6Thread
> + 0x362
> 0x00007fe3f993fb8a      jni_CallStaticVoidMethod + 0x17a
> 0x00007fe3fa23abcc      JavaMain + 0x80c
> ----------------- 12839 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12840 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12841 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12842 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12843 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12844 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12845 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12846 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12847 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12848 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12849 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12850 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12851 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12852 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f971085b      _ZN24ConcurrentG1RefineThread21run_young_rs_samplingEv
> +
> 0x7b
> 0x00007fe3f9710f78      _ZN24ConcurrentG1RefineThread3runEv + 0x3f8
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12853 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12854 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12855 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12856 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12857 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12858 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12859 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12860 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12861 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12862 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12863 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12864 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12865 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9710d8f      _ZN24ConcurrentG1RefineThread3runEv + 0x20f
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12866 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f973de27      _ZN20ConcurrentMarkThread3runEv + 0xc7
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12867 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12868 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12869 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12870 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d2fbeb      _ZN10GangWorker4loopEv + 0x5b
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12896 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9d174f9      _ZN8VMThread4loopEv + 0x339
> 0x00007fe3f9d17800      _ZN8VMThread3runEv + 0x70
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12897 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b5dff5      _ZN13ObjectMonitor4waitElbP6Thread + 0xa65
> 0x00007fe3f997d3b5      JVM_MonitorWait + 0x1a5
> 0x00007fe3e6039ce8      <Unknown compiled code>
> 0x00007fe3e58c9904      * java.lang.Object.wait() bci:2 line:502 (Compiled
> frame)
> * java.lang.ref.Reference$ReferenceHandler.run() bci:36 line:157
> (Interpreted frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12898 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b5dff5      _ZN13ObjectMonitor4waitElbP6Thread + 0xa65
> 0x00007fe3f997d3b5      JVM_MonitorWait + 0x1a5
> 0x00007fe3e6039ce8      <Unknown compiled code>
> ----------------- 12899 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f971114e      _ZN21SurrogateLockerThread4loopEv + 0x4e
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12900 -----------------
> 0x000000311860da00      sem_wait + 0x30
> 0x00007fe3f9b68f63      _ZL19signal_thread_entryP10JavaThreadP6Thread +
> 0x73
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12901 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12902 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12903 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12904 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12905 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12906 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12907 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12908 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12909 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12910 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12911 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12912 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d7c6      _ZN7Monitor4waitEblb + 0x2d6
> 0x00007fe3f9701c7b      _ZN12CompileQueue3getEv + 0x13b
> 0x00007fe3f9703ecb      _ZN13CompileBroker20compiler_thread_loopEv + 0x18b
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12913 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b2ce27      _ZN7Monitor5IWaitEP6Threadl + 0xf7
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9c0a7a8
> _ZN13ServiceThread20service_thread_entryEP10JavaThreadP6Thread + 0x398
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12916 -----------------
> 0x00000031182df113      __libc_poll + 0x53
> 0x00007fe331f68947      Java_java_net_PlainSocketImpl_socketAccept + 0x1e7
> 0x00007fe3e5015754      *
> java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) bci:0
> (Interpreted frame)
> 0x00007fe3e500798d      *
> java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) bci:7
> line:409
> (Interpreted frame)
> 0x00007fe3e500798d      * java.net.ServerSocket.
> implAccept(java.net.Socket)
> bci:60 line:545 (Interpreted frame)
> 0x00007fe3e500798d      * java.net.ServerSocket.accept() bci:48 line:513
> (Interpreted frame)
> 0x00007fe3e5007710      *
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() bci:55
> line:400 (Interpreted frame)
> 0x00007fe3e500798d      * sun.rmi.transport.tcp.
> TCPTransport$AcceptLoop.run()
> bci:1 line:372 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12917 -----------------
> 0x00000031182df113      __libc_poll + 0x53
> 0x00007fe331f68947      Java_java_net_PlainSocketImpl_socketAccept + 0x1e7
> 0x00007fe3e5015754      *
> java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) bci:0
> (Interpreted frame)
> 0x00007fe3e500798d      *
> java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) bci:7
> line:409
> (Interpreted frame)
> 0x00007fe3e500798d      * java.net.ServerSocket.
> implAccept(java.net.Socket)
> bci:60 line:545 (Interpreted frame)
> 0x00007fe3e500798d      * java.net.ServerSocket.accept() bci:48 line:513
> (Interpreted frame)
> 0x00007fe3e5007710      *
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() bci:55
> line:400 (Interpreted frame)
> 0x00007fe3e500798d      * sun.rmi.transport.tcp.
> TCPTransport$AcceptLoop.run()
> bci:1 line:372 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12918 -----------------
> 0x00000031182df113      __libc_poll + 0x53
> 0x00007fe331f68947      Java_java_net_PlainSocketImpl_socketAccept + 0x1e7
> 0x00007fe3e5015754      *
> java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) bci:0
> (Interpreted frame)
> 0x00007fe3e500798d      *
> java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) bci:7
> line:409
> (Interpreted frame)
> 0x00007fe3e500798d      * java.net.ServerSocket.
> implAccept(java.net.Socket)
> bci:60 line:545 (Interpreted frame)
> 0x00007fe3e500798d      * java.net.ServerSocket.accept() bci:48 line:513
> (Interpreted frame)
> 0x00007fe3e5007710      *
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() bci:1
> line:52 (Interpreted frame)
> 0x00007fe3e5007710      *
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() bci:55
> line:400 (Interpreted frame)
> 0x00007fe3e500798d      * sun.rmi.transport.tcp.
> TCPTransport$AcceptLoop.run()
> bci:1 line:372 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12919 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b2d17e      _ZN7Monitor5IWaitEP6Threadl + 0x44e
> 0x00007fe3f9b2d746      _ZN7Monitor4waitEblb + 0x256
> 0x00007fe3f9cb86e1      _ZNK13WatcherThread5sleepEv + 0x71
> 0x00007fe3f9cb9c9e      _ZN13WatcherThread3runEv + 0x5e
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12921 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5015754      * sun.misc.Unsafe.park(boolean, long) bci:0
> (Interpreted
> frame)
> 0x00007fe3e500798d      * java.util.concurrent.locks.LockSupport.park()
> bci:5
> line:304 (Interpreted frame)
> 0x00007fe3e500798d      *
> org.apache.ignite.internal.util.StripedExecutor$
> StripeConcurrentQueue.take()
> bci:65 line:581 (Interpreted frame)
> 0x00007fe3e5007710      *
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run() bci:8
> line:477
> (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12922 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12923 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12924 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12925 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12926 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12927 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12928 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12929 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12930 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12931 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12932 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12933 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12934 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12935 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12936 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12937 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12938 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12939 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12940 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12941 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12942 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12943 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5015754      * sun.misc.Unsafe.park(boolean, long) bci:0
> (Interpreted
> frame)
> 0x00007fe3e500798d      * java.util.concurrent.locks.LockSupport.park()
> bci:5
> line:304 (Interpreted frame)
> 0x00007fe3e500798d      *
> org.apache.ignite.internal.util.StripedExecutor$
> StripeConcurrentQueue.take()
> bci:65 line:581 (Interpreted frame)
> 0x00007fe3e5007710      *
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run() bci:8
> line:477
> (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12944 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12945 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12946 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5015754      * sun.misc.Unsafe.park(boolean, long) bci:0
> (Interpreted
> frame)
> 0x00007fe3e500798d      * java.util.concurrent.locks.LockSupport.park()
> bci:5
> line:304 (Interpreted frame)
> 0x00007fe3e500798d      *
> org.apache.ignite.internal.util.StripedExecutor$
> StripeConcurrentQueue.take()
> bci:65 line:581 (Interpreted frame)
> 0x00007fe3e5007710      *
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run() bci:8
> line:477
> (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12947 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12948 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12949 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12950 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12951 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12952 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12953 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12954 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12955 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12956 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12957 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12958 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12959 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12960 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12961 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12962 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12963 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12964 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12965 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12966 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12967 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12968 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12969 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12970 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12971 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12972 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12973 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12974 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5015754      * sun.misc.Unsafe.park(boolean, long) bci:0
> (Interpreted
> frame)
> 0x00007fe3e57cf6d4      * java.util.concurrent.locks.LockSupport.park()
> bci:5
> line:304 (Compiled frame)
> *
> org.apache.ignite.internal.util.StripedExecutor$
> StripeConcurrentQueue.take()
> bci:65 line:581 (Compiled frame)
> * org.apache.ignite.internal.util.StripedExecutor$Stripe.run() bci:8
> line:477 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12975 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12976 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12977 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12978 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12979 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12980 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5015754      * sun.misc.Unsafe.park(boolean, long) bci:0
> (Interpreted
> frame)
> 0x00007fe3e57cf6d4      * java.util.concurrent.locks.LockSupport.park()
> bci:5
> line:304 (Compiled frame)
> *
> org.apache.ignite.internal.util.StripedExecutor$
> StripeConcurrentQueue.take()
> bci:65 line:581 (Compiled frame)
> * org.apache.ignite.internal.util.StripedExecutor$Stripe.run() bci:8
> line:477 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 12981 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12982 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12983 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12984 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12991 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b72f13      _ZN2os5sleepEP6Threadlb + 0x283
> 0x00007fe3f9974e22      JVM_Sleep + 0x3b2
> 0x00007fe3e5a01db1      <Unknown compiled code>
> ----------------- 12992 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b5d80f      _ZN13ObjectMonitor4waitElbP6Thread + 0x27f
> 0x00007fe3f997d3b5      JVM_MonitorWait + 0x1a5
> 0x00007fe3e6039ce8      <Unknown compiled code>
> ----------------- 12995 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 12996 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 12997 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 12998 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 12999 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13000 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13001 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13002 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13003 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13004 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13005 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13006 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13007 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13008 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13009 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13010 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b72f13      _ZN2os5sleepEP6Threadlb + 0x283
> 0x00007fe3f9974e22      JVM_Sleep + 0x3b2
> 0x00007fe3e5a01db1      <Unknown compiled code>
> 0x00007fe3e794402c      * it.sauronsoftware.cron4j.
> TimerThread.safeSleep(long)
> bci:10 (Compiled frame)
> * it.sauronsoftware.cron4j.TimerThread.run() bci:33 (Interpreted frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13011 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13012 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13013 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13014 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13015 -----------------
> 0x00000031182e8f33      __GI_epoll_wait + 0x33
> 0x00007fe3e566733a      <Unknown compiled code>
> ----------------- 13016 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5015754      * sun.misc.Unsafe.park(boolean, long) bci:0
> (Interpreted
> frame)
> 0x00007fe3e500798d      *
> java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14
> line:175 (Interpreted frame)
> 0x00007fe3e500798d      *
> java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await()
> bci:42 line:2039 (Interpreted frame)
> 0x00007fe3e50079d2      * java.util.concurrent.DelayQueue.take() bci:28
> line:211
> (Interpreted frame)
> 0x00007fe3e5007710      *
> org.apache.ignite.internal.processors.datastreamer.
> DataStreamProcessor$2.body()
> bci:14 line:108 (Interpreted frame)
> 0x00007fe3e500798d      * org.apache.ignite.internal.
> util.worker.GridWorker.run()
> bci:82 line:110 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13017 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13018 -----------------
> 0x00000031182df113      __libc_poll + 0x53
> 0x00007fe331f68947      Java_java_net_PlainSocketImpl_socketAccept + 0x1e7
> 0x00007fe3e5015754      *
> java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) bci:0
> (Interpreted frame)
> 0x00007fe3e500798d      *
> java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) bci:7
> line:409
> (Interpreted frame)
> 0x00007fe3e500798d      * java.net.ServerSocket.
> implAccept(java.net.Socket)
> bci:60 line:545 (Interpreted frame)
> 0x00007fe3e500798d      * java.net.ServerSocket.accept() bci:48 line:513
> (Interpreted frame)
> 0x00007fe3e5007710      *
> org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body() bci:11
> line:5592 (Interpreted frame)
> 0x00007fe3e500798d      * org.apache.ignite.spi.IgniteSpiThread.run()
> bci:1
> line:62 (Interpreted frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13019 -----------------
> 0x000000311860ec2c      __libc_recv + 0x6c
> 0x00007fe331f69b3b      Java_java_net_SocketInputStream_socketRead0 + 0x9b
> 0x00007fe3e52db5e9      <Unknown compiled code>
> ----------------- 13021 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13022 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13023 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b72f13      _ZN2os5sleepEP6Threadlb + 0x283
> 0x00007fe3f9974e22      JVM_Sleep + 0x3b2
> 0x00007fe3e5a01db1      <Unknown compiled code>
> ----------------- 13059 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9b5dff5      _ZN13ObjectMonitor4waitElbP6Thread + 0xa65
> 0x00007fe3f997d3b5      JVM_MonitorWait + 0x1a5
> 0x00007fe3e6039ce8      <Unknown compiled code>
> 0x00007fe3e58c9904      * java.lang.Object.wait() bci:2 line:502 (Compiled
> frame)
> *
> org.apache.ignite.internal.processors.cache.persistence.
> wal.FileWriteAheadLogManager$FileArchiver.run()
> bci:202 line:1204 (Interpreted frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13062 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b72f13      _ZN2os5sleepEP6Threadlb + 0x283
> 0x00007fe3f9974e22      JVM_Sleep + 0x3b2
> 0x00007fe3e5a01db1      <Unknown compiled code>
> ----------------- 13064 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9b5d80f      _ZN13ObjectMonitor4waitElbP6Thread + 0x27f
> 0x00007fe3f997d3b5      JVM_MonitorWait + 0x1a5
> 0x00007fe3e6039ce8      <Unknown compiled code>
> 0x00007fe3e6bd0614      *
> org.apache.ignite.internal.processors.cache.persistence.
> GridCacheDatabaseSharedManager$Checkpointer.waitCheckpointEvent()
> bci:38 line:2069 (Compiled frame)
> *
> org.apache.ignite.internal.processors.cache.persistence.
> GridCacheDatabaseSharedManager$Checkpointer.body()
> bci:8 line:1835 (Interpreted frame)
> 0x00007fe3e500798d      * org.apache.ignite.internal.
> util.worker.GridWorker.run()
> bci:82 line:110 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13086 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13155 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> 0x00007fe3e60f433c      *
> java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14
> line:175 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt()
> bci:1 line:836 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(int)
> bci:72 line:997 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(int)
> bci:24 line:1304 (Compiled frame)
> * java.util.concurrent.CountDownLatch.await() bci:5 line:231 (Compiled
> frame)
> * org.apache.ignite.internal.processors.schedule.ScheduleFutureImpl.get()
> bci:10 line:703 (Compiled frame)
> *
> com.calm.report.service.AbstractExpiryService.
> afterExpiry(org.apache.ignite.scheduler.SchedulerFuture,
> org.apache.ignite.services.ServiceContext) bci:75 line:49 (Interpreted
> frame)
> 0x00007fe3e500798d      *
> com.calm.report.service.TradeItemExpiryService.execute(org.apache.ignite.
> services.ServiceContext)
> bci:28 line:72 (Interpreted frame)
> 0x00007fe3e50079d2      *
> org.apache.ignite.internal.processors.service.GridServiceProcessor$2.run()
> bci:8 line:1160 (Interpreted frame)
> 0x00007fe3e50079d2      *
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.
> ThreadPoolExecutor$Worker)
> bci:95 line:1142 (Interpreted frame)
> 0x00007fe3e500798d      * java.util.concurrent.
> ThreadPoolExecutor$Worker.run()
> bci:5 line:617 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13182 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> 0x00007fe3e60f433c      *
> java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14
> line:175 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt()
> bci:1 line:836 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(int)
> bci:72 line:997 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(int)
> bci:24 line:1304 (Compiled frame)
> * java.util.concurrent.CountDownLatch.await() bci:5 line:231 (Compiled
> frame)
> * org.apache.ignite.internal.processors.schedule.ScheduleFutureImpl.get()
> bci:10 line:703 (Compiled frame)
> *
> com.calm.report.service.AbstractExpiryService.
> afterExpiry(org.apache.ignite.scheduler.SchedulerFuture,
> org.apache.ignite.services.ServiceContext) bci:75 line:49 (Interpreted
> frame)
> 0x00007fe3e500798d      *
> com.calm.report.service.TradeCustomerExpiryService.
> execute(org.apache.ignite.services.ServiceContext)
> bci:28 line:68 (Interpreted frame)
> 0x00007fe3e50079d2      *
> org.apache.ignite.internal.processors.service.GridServiceProcessor$2.run()
> bci:8 line:1160 (Interpreted frame)
> 0x00007fe3e50079d2      *
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.
> ThreadPoolExecutor$Worker)
> bci:95 line:1142 (Interpreted frame)
> 0x00007fe3e500798d      * java.util.concurrent.
> ThreadPoolExecutor$Worker.run()
> bci:5 line:617 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13195 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> 0x00007fe3e60f433c      *
> java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14
> line:175 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt()
> bci:1 line:836 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(int)
> bci:72 line:997 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(int)
> bci:24 line:1304 (Compiled frame)
> * java.util.concurrent.CountDownLatch.await() bci:5 line:231 (Compiled
> frame)
> * org.apache.ignite.internal.processors.schedule.ScheduleFutureImpl.get()
> bci:10 line:703 (Compiled frame)
> *
> com.calm.report.service.AbstractExpiryService.
> afterExpiry(org.apache.ignite.scheduler.SchedulerFuture,
> org.apache.ignite.services.ServiceContext) bci:75 line:49 (Interpreted
> frame)
> 0x00007fe3e500798d      *
> com.calm.report.service.TradeExtraExpiryService.execute(org.apache.ignite.
> services.ServiceContext)
> bci:28 line:63 (Interpreted frame)
> 0x00007fe3e50079d2      *
> org.apache.ignite.internal.processors.service.GridServiceProcessor$2.run()
> bci:8 line:1160 (Interpreted frame)
> 0x00007fe3e50079d2      *
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.
> ThreadPoolExecutor$Worker)
> bci:95 line:1142 (Interpreted frame)
> 0x00007fe3e500798d      * java.util.concurrent.
> ThreadPoolExecutor$Worker.run()
> bci:5 line:617 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13202 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> 0x00007fe3e60f433c      *
> java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14
> line:175 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt()
> bci:1 line:836 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(int)
> bci:72 line:997 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(int)
> bci:24 line:1304 (Compiled frame)
> * java.util.concurrent.CountDownLatch.await() bci:5 line:231 (Compiled
> frame)
> * org.apache.ignite.internal.processors.schedule.ScheduleFutureImpl.get()
> bci:10 line:703 (Compiled frame)
> *
> com.calm.report.service.AbstractExpiryService.
> afterExpiry(org.apache.ignite.scheduler.SchedulerFuture,
> org.apache.ignite.services.ServiceContext) bci:75 line:49 (Interpreted
> frame)
> 0x00007fe3e500798d      *
> com.calm.report.service.TradeItemExtraExpiryService.
> execute(org.apache.ignite.services.ServiceContext)
> bci:28 line:63 (Interpreted frame)
> 0x00007fe3e50079d2      *
> org.apache.ignite.internal.processors.service.GridServiceProcessor$2.run()
> bci:8 line:1160 (Interpreted frame)
> 0x00007fe3e50079d2      *
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.
> ThreadPoolExecutor$Worker)
> bci:95 line:1142 (Interpreted frame)
> 0x00007fe3e500798d      * java.util.concurrent.
> ThreadPoolExecutor$Worker.run()
> bci:5 line:617 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13210 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> 0x00007fe3e60f433c      *
> java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14
> line:175 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt()
> bci:1 line:836 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(int)
> bci:72 line:997 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(int)
> bci:24 line:1304 (Compiled frame)
> * java.util.concurrent.CountDownLatch.await() bci:5 line:231 (Compiled
> frame)
> * org.apache.ignite.internal.processors.schedule.ScheduleFutureImpl.get()
> bci:10 line:703 (Compiled frame)
> *
> com.calm.report.service.AbstractExpiryService.
> afterExpiry(org.apache.ignite.scheduler.SchedulerFuture,
> org.apache.ignite.services.ServiceContext) bci:75 line:49 (Interpreted
> frame)
> 0x00007fe3e500798d      *
> com.calm.report.service.TradePromotionExpiryService.
> execute(org.apache.ignite.services.ServiceContext)
> bci:28 line:63 (Interpreted frame)
> 0x00007fe3e50079d2      *
> org.apache.ignite.internal.processors.service.GridServiceProcessor$2.run()
> bci:8 line:1160 (Interpreted frame)
> 0x00007fe3e50079d2      *
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.
> ThreadPoolExecutor$Worker)
> bci:95 line:1142 (Interpreted frame)
> 0x00007fe3e500798d      * java.util.concurrent.
> ThreadPoolExecutor$Worker.run()
> bci:5 line:617 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13216 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> 0x00007fe3e60f433c      *
> java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14
> line:175 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt()
> bci:1 line:836 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(int)
> bci:72 line:997 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(int)
> bci:24 line:1304 (Compiled frame)
> * java.util.concurrent.CountDownLatch.await() bci:5 line:231 (Compiled
> frame)
> * org.apache.ignite.internal.processors.schedule.ScheduleFutureImpl.get()
> bci:10 line:703 (Compiled frame)
> *
> com.calm.report.service.AbstractExpiryService.
> afterExpiry(org.apache.ignite.scheduler.SchedulerFuture,
> org.apache.ignite.services.ServiceContext) bci:75 line:49 (Interpreted
> frame)
> 0x00007fe3e500798d      *
> com.calm.report.service.PaymentExpiryService.execute(
> org.apache.ignite.services.ServiceContext)
> bci:28 line:64 (Interpreted frame)
> 0x00007fe3e50079d2      *
> org.apache.ignite.internal.processors.service.GridServiceProcessor$2.run()
> bci:8 line:1160 (Interpreted frame)
> 0x00007fe3e50079d2      *
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.
> ThreadPoolExecutor$Worker)
> bci:95 line:1142 (Interpreted frame)
> 0x00007fe3e500798d      * java.util.concurrent.
> ThreadPoolExecutor$Worker.run()
> bci:5 line:617 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13217 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> 0x00007fe3e60f433c      *
> java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14
> line:175 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt()
> bci:1 line:836 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(int)
> bci:72 line:997 (Compiled frame)
> *
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(int)
> bci:24 line:1304 (Compiled frame)
> * java.util.concurrent.CountDownLatch.await() bci:5 line:231 (Compiled
> frame)
> * org.apache.ignite.internal.processors.schedule.ScheduleFutureImpl.get()
> bci:10 line:703 (Compiled frame)
> *
> com.calm.report.service.AbstractExpiryService.
> afterExpiry(org.apache.ignite.scheduler.SchedulerFuture,
> org.apache.ignite.services.ServiceContext) bci:75 line:49 (Interpreted
> frame)
> 0x00007fe3e500798d      *
> com.calm.report.service.PaymentItemExpiryService.
> execute(org.apache.ignite.services.ServiceContext)
> bci:28 line:63 (Interpreted frame)
> 0x00007fe3e50079d2      *
> org.apache.ignite.internal.processors.service.GridServiceProcessor$2.run()
> bci:8 line:1160 (Interpreted frame)
> 0x00007fe3e50079d2      *
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.
> ThreadPoolExecutor$Worker)
> bci:95 line:1142 (Interpreted frame)
> 0x00007fe3e500798d      * java.util.concurrent.
> ThreadPoolExecutor$Worker.run()
> bci:5 line:617 (Interpreted frame)
> 0x00007fe3e50079d2      * java.lang.Thread.run() bci:11 line:745
> (Interpreted
> frame)
> 0x00007fe3e50004e7      <StubRoutines>
> 0x00007fe3f98e1be6
> _ZN9JavaCalls11call_helperEP9JavaValueP12methodHan
> dleP17JavaCallArgumentsP6Thread
> + 0x1056
> 0x00007fe3f98e20f1
> _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_
> P17JavaCallArgumentsP6Thread
> + 0x321
> 0x00007fe3f98e2597
> _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11Kl
> assHandleP6SymbolS5_P6Thread
> + 0x47
> 0x00007fe3f99792d0      _ZL12thread_entryP10JavaThreadP6Thread + 0xa0
> 0x00007fe3f9cbef3f      _ZN10JavaThread17thread_main_innerEv + 0xdf
> 0x00007fe3f9cbf06c      _ZN10JavaThread3runEv + 0x11c
> 0x00007fe3f9b72b88      _ZL10java_startP6Thread + 0x108
> ----------------- 13418 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13419 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13420 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13421 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13422 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13423 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13424 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 13425 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 24236 -----------------
> ----------------- 25398 -----------------
> 0x00007fe3e66bd072      <Unknown compiled code>
> ----------------- 30073 -----------------
> 0x00007fe3e79cf689      <Unknown compiled code>
> ----------------- 30591 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 31434 -----------------
> 0x00007fe3e6e57d08      <Unknown compiled code>
> ----------------- 31436 -----------------
> 0x00007fe3e5052182      <StubRoutines>
> ----------------- 31540 -----------------
> 0x00007fe3e707daa5      * java.util.HashMap.getNode(int, java.lang.Object)
> bci:143 line:580 (Compiled frame)
> * java.util.HashMap.get(java.lang.Object) bci:6 line:556 (Compiled frame)
> * org.h2.expression.ExpressionColumn.getValue(org.h2.engine.Session)
> bci:25
> line:180 (Compiled frame)
> * org.h2.expression.Comparison.getValue(org.h2.engine.Session) bci:5
> line:237 (Compiled frame)
> ----------------- 31577 -----------------
> 0x00007fe3e6546e23      <Unknown compiled code>
> ----------------- 31785 -----------------
> 0x00007fe3e5051ee9      <StubRoutines>
> ----------------- 31877 -----------------
> 0x00007fe3e6731428      * org.h2.value.Value.getOrder(int) bci:235
> line:337
> (Compiled frame)
> * org.h2.value.Value.getHigherOrder(int, int) bci:55 line:366 (Compiled
> frame)
> * org.h2.table.Table.compareTypeSafe(org.h2.value.Value,
> org.h2.value.Value)
> bci:15 line:1165 (Compiled frame)
> *
> org.apache.ignite.internal.processors.query.h2.database.
> H2TreeIndex$1.compareValues(org.h2.value.Value,
> org.h2.value.Value) bci:18 line:123 (Compiled frame)
> *
> org.apache.ignite.internal.processors.query.h2.database.
> H2Tree$1.compare(org.h2.value.Value,
> org.h2.value.Value) bci:6 line:59 (Compiled frame)
> *
> org.apache.ignite.internal.processors.query.h2.database.
> H2Tree$1.compare(java.lang.Object,
> java.lang.Object) bci:9 line:57 (Compiled frame)
> *
> org.apache.ignite.internal.processors.query.h2.database.
> InlineIndexHelper.compare(long,
> int, int, org.h2.value.Value, java.util.Comparator) bci:24 line:338
> (Compiled frame)
> *
> org.apache.ignite.internal.processors.query.h2.database.
> H2Tree.compare(org.apache.ignite.internal.processors.
> cache.persistence.tree.io.BPlusIO,
> long, int, org.h2.result.SearchRow) bci:114 line:180 (Compiled frame)
> ----------------- 31879 -----------------
> 0x00007fe3e63b9ade      <Unknown compiled code>
> ----------------- 31911 -----------------
> 0x00007fe3e648fb32      <Unknown compiled code>
> ----------------- 31930 -----------------
> 0x000000311860b68c      __pthread_cond_wait + 0xcc
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 32008 -----------------
> 0x00007fe3e5051ef2      <StubRoutines>
> ----------------- 28481 -----------------
> 0x00007fe3e73e54e4      *
> org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache.nodes(int,
> org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion)
> bci:2 line:407 (Compiled frame)
> *
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.
> nodesByPartition(int,
> org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion)
> bci:59 line:220 (Compiled frame)
> *
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.
> primaryByPartition(int,
> org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion)
> bci:3 line:256 (Compiled frame)
> *
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.
> primaryByKey(java.lang.Object,
> org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion)
> bci:7 line:247 (Compiled frame)
> *
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.
> primaryByKey(org.apache.ignite.cluster.ClusterNode,
> java.lang.Object,
> org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion)
> bci:3 line:271 (Compiled frame)
> ----------------- 32723 -----------------
> 0x00007fe3e5ae5393      <Unknown compiled code>
> ----------------- 1779 -----------------
> 0x00007fe3e6546e23      <Unknown compiled code>
> ----------------- 2023 -----------------
> 0x00007fe3e661d5c5      <Unknown compiled code>
> ----------------- 2508 -----------------
> 0x00007fe3e7dca85e      *
> org.apache.ignite.internal.processors.query.h2.opt.
> GridH2AbstractKeyValueRow.<init>(org.apache.ignite.
> internal.processors.query.h2.opt.GridH2RowDescriptor,
> java.lang.Object, int, java.lang.Object, int,
> org.apache.ignite.internal.processors.cache.version.GridCacheVersion,
> long)
> bci:60 line:96 (Compiled frame)
> *
> org.apache.ignite.internal.processors.query.h2.opt.
> GridH2KeyValueRowOnheap.<init>(org.apache.ignite.internal.
> processors.query.h2.opt.GridH2RowDescriptor,
> java.lang.Object, int, java.lang.Object, int,
> org.apache.ignite.internal.processors.cache.version.GridCacheVersion,
> long)
> bci:12 line:44 (Compiled frame)
> *
> org.apache.ignite.internal.processors.query.h2.
> H2RowDescriptor.createRow(org.apache.ignite.internal.processors.cache.
> KeyCacheObject,
> int, org.apache.ignite.internal.processors.cache.CacheObject,
> org.apache.ignite.internal.processors.cache.version.GridCacheVersion,
> long)
> bci:50 line:307 (Compiled frame)
> ----------------- 2695 -----------------
> 0x00007fe3e6e3dd06      *
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.
> partition(java.lang.Object,
> boolean) bci:79 line:180 (Compiled frame)
> *
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.
> partition(java.lang.Object)
> bci:3 line:155 (Compiled frame)
> *
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.
> primaryByKey(java.lang.Object,
> org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion)
> bci:3 line:247 (Compiled frame)
> *
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.
> primaryByKey(org.apache.ignite.cluster.ClusterNode,
> java.lang.Object,
> org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion)
> bci:3 line:271 (Compiled frame)
> ----------------- 3259 -----------------
> 0x00007fe3e5e02143      <Unknown compiled code>
> ----------------- 4208 -----------------
> 0x00007fe3e5051fe4      <StubRoutines>
> ----------------- 29481 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 29547 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 29602 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 29605 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 29650 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 29692 -----------------
> 0x000000311860ba5e      __pthread_cond_timedwait + 0x13e
> 0x00007fe3f9ce5355      Unsafe_Park + 0xf5
> 0x00007fe3e5a01a2a      <Unknown compiled code>
> ----------------- 12837 -----------------
> 0x00000031186082fd      pthread_join + 0x10d
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Mime
View raw message