hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edson Ramiro <erlfi...@gmail.com>
Subject Re: Jetty can't start the SelectChannelConnector
Date Tue, 06 Apr 2010 23:05:08 GMT
ok,


[erlf07@cohiba ~ ]$ cat java_native_frames_dn
Attaching to process ID 3836, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.2-b01
Deadlock Detection:

No deadlocks found.

----------------- 3864 -----------------
0xf775a4f1    __libc_read + 0x41
0xf702298c    readBytes + 0xdc
0xf701e717    Java_java_io_FileInputStream_readBytes + 0x47
0xf400b4aa    * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
(Interpreted frame)
0xf4003f69    * java.io.FileInputStream.read(byte[], int, int) bci:4
line:199 (Interpreted frame)
0xf4003f69    * java.io.BufferedInputStream.read1(byte[], int, int) bci:39
line:256 (Interpreted frame)
0xf4003f69    * java.io.BufferedInputStream.read(byte[], int, int) bci:49
line:317 (Interpreted frame)
0xf4003f69    * java.io.BufferedInputStream.fill() bci:175 line:218
(Interpreted frame)
0xf400408d    * java.io.BufferedInputStream.read1(byte[], int, int) bci:44
line:258 (Interpreted frame)
0xf4003f69    * java.io.BufferedInputStream.read(byte[], int, int) bci:49
line:317 (Interpreted frame)
0xf4003f69    *
sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
line:453 (Interpreted frame)
0xf4003e61    * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
bci:11 line:123 (Interpreted frame)
0xf400408d    * sun.security.provider.SeedGenerator.generateSeed(byte[])
bci:4 line:118 (Interpreted frame)
0xf400408d    * sun.security.provider.SecureRandom.engineGenerateSeed(int)
bci:5 line:114 (Interpreted frame)
0xf4003f27    * sun.security.provider.SecureRandom.engineNextBytes(byte[])
bci:40 line:171 (Interpreted frame)
0xf400408d    * java.security.SecureRandom.nextBytes(byte[]) bci:5 line:433
(Interpreted frame)
0xf400408d    * java.security.SecureRandom.next(int) bci:17 line:455
(Interpreted frame)
0xf4003f69    * java.util.Random.nextLong() bci:3 line:284 (Interpreted
frame)
0xf4003fab    * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
bci:73 line:139 (Interpreted frame)
0xf400408d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569    * org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
bci:96 line:168 (Interpreted frame)
0xf400408d    * org.mortbay.jetty.servlet.HashSessionManager.doStart()
bci:12 line:67 (Interpreted frame)
0xf400408d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569    * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
line:115 (Interpreted frame)
0xf400408d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569    * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
line:130 (Interpreted frame)
0xf400408d    * org.mortbay.jetty.handler.ContextHandler.startContext()
bci:1 line:537 (Interpreted frame)
0xf400408d    * org.mortbay.jetty.servlet.Context.startContext() bci:1
line:136 (Interpreted frame)
0xf400408d    * org.mortbay.jetty.webapp.WebAppContext.startContext()
bci:123 line:1234 (Interpreted frame)
0xf400408d    * org.mortbay.jetty.handler.ContextHandler.doStart() bci:140
line:517 (Interpreted frame)
0xf400408d    * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
line:460 (Interpreted frame)
0xf400408d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569    * org.mortbay.jetty.handler.HandlerCollection.doStart() bci:32
line:152 (Interpreted frame)
0xf400408d    * org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
bci:5 line:156 (Interpreted frame)
0xf400408d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf4004569    * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
line:130 (Interpreted frame)
0xf400408d    * org.mortbay.jetty.Server.doStart() bci:201 line:222
(Interpreted frame)
0xf400408d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf400408d    * org.apache.hadoop.http.HttpServer.start() bci:383 line:461
(Interpreted frame)
0xf400408d    *
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
java.util.AbstractList) bci:916 line:375 (Interpreted frame)
0xf400408d    *
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
java.util.AbstractList) bci:158 line:216 (Interpreted frame)
0xf400408d    *
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
org.apache.hadoop.conf.Configuration) bci:96 line:1283 (Interpreted frame)
0xf4003f27    *
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
org.apache.hadoop.conf.Configuration) bci:92 line:1238 (Interpreted frame)
0xf4003f27    *
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
org.apache.hadoop.conf.Configuration) bci:2 line:1246 (Interpreted frame)
0xf4003f27    *
org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
bci:12 line:1368 (Interpreted frame)
0xf400134c    <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x06375b30
_ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread +
0x20
0x0639f2e6
_Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
+ 0x116
0x06390ae8    jni_CallStaticVoidMethod + 0xf8
0x08049b98    JavaMain + 0x2c8
0xf7753f50    start_thread + 0xa0
----------------- 3865 -----------------
0xf77578d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071    _ZN7Monitor4waitEblb + 0x221
0x063272f6    _ZN13GCTaskManager8get_taskEj + 0x66
0x0632843b    _ZN12GCTaskThread3runEv + 0xdb
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3866 -----------------
0xf77578d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071    _ZN7Monitor4waitEblb + 0x221
0x063272f6    _ZN13GCTaskManager8get_taskEj + 0x66
0x0632843b    _ZN12GCTaskThread3runEv + 0xdb
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3869 -----------------
0xf7757be8    __pthread_cond_timedwait + 0xe8
0x39d73c58    ????????
sun.jvm.hotspot.debugger.UnmappedAddressException
    at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
    at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
    at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
    at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
    at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 3870 -----------------
0xf77578d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x065a1a31    _ZN13ObjectMonitor4waitExbP6Thread + 0x571
0x0659f3a2    _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
0x063c834a    JVM_MonitorWait + 0x1da
0xf400b4aa    * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xf400408d    * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0xf400408d    * java.lang.ref.Reference$ReferenceHandler.run() bci:46
line:116 (Interpreted frame)
0xf400134c    <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5    _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3871 -----------------
0xf77578d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x065a1a31    _ZN13ObjectMonitor4waitExbP6Thread + 0x571
0x0659f3a2    _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
0x063c834a    JVM_MonitorWait + 0x1da
0xf400b4aa    * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xf400408d    * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
(Interpreted frame)
0xf4003f27    * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
(Interpreted frame)
0xf4003f27    * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159
(Interpreted frame)
0xf400134c    <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5    _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3872 -----------------
0xf77598b0    __new_sem_wait + 0x40
0x06503c0d    _ZN2os11signal_waitEv + 0xd
0x06500d28    _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3873 -----------------
0xf77578d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064edf21    _ZN7Monitor4waitEblb + 0xd1
0x062ada2f    _ZN12CompileQueue3getEv + 0x5f
0x062af2b4    _ZN13CompileBroker20compiler_thread_loopEv + 0x184
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3874 -----------------
0xf77578d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064edf21    _ZN7Monitor4waitEblb + 0xd1
0x062ada2f    _ZN12CompileQueue3getEv + 0x5f
0x062af2b4    _ZN13CompileBroker20compiler_thread_loopEv + 0x184
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3875 -----------------
0xf77578d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071    _ZN7Monitor4waitEblb + 0x221
0x064bc935
_ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
+ 0x85
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3878 -----------------
0xf775a6fe    __libc_accept + 0x3e
0xb0c41a19    Java_java_net_PlainSocketImpl_socketAccept + 0x129
0xf400b4aa    * java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl)
bci:0 (Interpreted frame)
0xf400408d    * java.net.PlainSocketImpl.accept(java.net.SocketImpl) bci:7
line:384 (Interpreted frame)
0xf400408d    * java.net.ServerSocket.implAccept(java.net.Socket) bci:60
line:453 (Interpreted frame)
0xf400408d    * java.net.ServerSocket.accept() bci:48 line:421 (Interpreted
frame)
0xf4003f27    *
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() bci:1
line:34 (Interpreted frame)
0xf4003f27    *
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() bci:55
line:369 (Interpreted frame)
0xf400408d    * sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() bci:1
line:341 (Interpreted frame)
0xf4004569    * java.lang.Thread.run() bci:11 line:619 (Interpreted frame)
0xf400134c    <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5    _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3879 -----------------
0xf7757be8    __pthread_cond_timedwait + 0xe8
0x1bef7958    ????????
sun.jvm.hotspot.debugger.UnmappedAddressException
    at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
    at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
    at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
    at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
    at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 4055 -----------------
0xf7757be8    __pthread_cond_timedwait + 0xe8
0x14e31390    ????????
sun.jvm.hotspot.debugger.UnmappedAddressException
    at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
    at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
    at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
    at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
    at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 4060 -----------------
0xf7757be8    __pthread_cond_timedwait + 0xe8
0x358d0f88    ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
address: 0x4bbbaf53 with alignment: 4
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
    at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
    at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
    at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 4062 -----------------
0xf7757be8    __pthread_cond_timedwait + 0xe8
0x35c10c20    ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
address: 0x4bbbaf53 with alignment: 4
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
    at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
    at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
    at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 7801 -----------------
0xf775a6fe    __libc_accept + 0x3e
0x061c1a78    _ZN14AttachListener7dequeueEv + 0x68
0x061c0e55    _Z28attach_listener_thread_entryP10JavaThreadP6Thread + 0x35
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf7753f50    start_thread + 0xa0
----------------- 3836 -----------------
0xf7754452    pthread_join + 0x112
0x0804dd78    ContinueInNewThread + 0x88
0x080497f6    main + 0x276
0xf7614775    ????????



[erlf07@cohiba ~ ]$ cat java_native_frames_tt
Attaching to process ID 22510, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.2-b01
Deadlock Detection:

No deadlocks found.

----------------- 22544 -----------------
0xf76e34f1    __libc_read + 0x41
0xf6fab98c    readBytes + 0xdc
0xf6fa7717    Java_java_io_FileInputStream_readBytes + 0x47
0xf3f944aa    * java.io.FileInputStream.readBytes(byte[], int, int) bci:0
(Interpreted frame)
0xf3f8cf69    * java.io.FileInputStream.read(byte[], int, int) bci:4
line:199 (Interpreted frame)
0xf3f8cf69    * java.io.BufferedInputStream.read1(byte[], int, int) bci:39
line:256 (Interpreted frame)
0xf3f8cf69    * java.io.BufferedInputStream.read(byte[], int, int) bci:49
line:317 (Interpreted frame)
0xf3f8cf69    * java.io.BufferedInputStream.fill() bci:175 line:218
(Interpreted frame)
0xf3f8d08d    * java.io.BufferedInputStream.read1(byte[], int, int) bci:44
line:258 (Interpreted frame)
0xf3f8cf69    * java.io.BufferedInputStream.read(byte[], int, int) bci:49
line:317 (Interpreted frame)
0xf3f8cf69    *
sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12
line:453 (Interpreted frame)
0xf3f8ce61    * sun.security.provider.SeedGenerator.getSeedBytes(byte[])
bci:11 line:123 (Interpreted frame)
0xf3f8d08d    * sun.security.provider.SeedGenerator.generateSeed(byte[])
bci:4 line:118 (Interpreted frame)
0xf3f8d08d    * sun.security.provider.SecureRandom.engineGenerateSeed(int)
bci:5 line:114 (Interpreted frame)
0xf3f8cf27    * sun.security.provider.SecureRandom.engineNextBytes(byte[])
bci:40 line:171 (Interpreted frame)
0xf3f8d08d    * java.security.SecureRandom.nextBytes(byte[]) bci:5 line:433
(Interpreted frame)
0xf3f8d08d    * java.security.SecureRandom.next(int) bci:17 line:455
(Interpreted frame)
0xf3f8cf69    * java.util.Random.nextLong() bci:3 line:284 (Interpreted
frame)
0xf3f8cfab    * org.mortbay.jetty.servlet.HashSessionIdManager.doStart()
bci:73 line:139 (Interpreted frame)
0xf3f8d08d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569    * org.mortbay.jetty.servlet.AbstractSessionManager.doStart()
bci:96 line:168 (Interpreted frame)
0xf3f8d08d    * org.mortbay.jetty.servlet.HashSessionManager.doStart()
bci:12 line:67 (Interpreted frame)
0xf3f8d08d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569    * org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4
line:115 (Interpreted frame)
0xf3f8d08d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569    * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
line:130 (Interpreted frame)
0xf3f8d08d    * org.mortbay.jetty.handler.ContextHandler.startContext()
bci:1 line:537 (Interpreted frame)
0xf3f8d08d    * org.mortbay.jetty.servlet.Context.startContext() bci:1
line:136 (Interpreted frame)
0xf3f8d08d    * org.mortbay.jetty.webapp.WebAppContext.startContext()
bci:123 line:1234 (Interpreted frame)
0xf3f8d08d    * org.mortbay.jetty.handler.ContextHandler.doStart() bci:140
line:517 (Interpreted frame)
0xf3f8d08d    * org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170
line:460 (Interpreted frame)
0xf3f8d08d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569    * org.mortbay.jetty.handler.HandlerCollection.doStart() bci:32
line:152 (Interpreted frame)
0xf3f8d08d    * org.mortbay.jetty.handler.ContextHandlerCollection.doStart()
bci:5 line:156 (Interpreted frame)
0xf3f8d08d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d569    * org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11
line:130 (Interpreted frame)
0xf3f8d08d    * org.mortbay.jetty.Server.doStart() bci:201 line:222
(Interpreted frame)
0xf3f8d08d    * org.mortbay.component.AbstractLifeCycle.start() bci:31
line:50 (Interpreted frame)
0xf3f8d08d    * org.apache.hadoop.http.HttpServer.start() bci:383 line:461
(Interpreted frame)
0xf3f8d08d    *
org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
bci:519 line:931 (Interpreted frame)
0xf3f8d08d    *
org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) bci:52
line:2833 (Interpreted frame)
0xf3f8a34c    <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x06375b30
_ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread +
0x20
0x0639f2e6
_Z17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread
+ 0x116
0x06390ae8    jni_CallStaticVoidMethod + 0xf8
0x08049b98    JavaMain + 0x2c8
0xf76dcf50    start_thread + 0xa0
----------------- 22545 -----------------
0xf76e08d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071    _ZN7Monitor4waitEblb + 0x221
0x063272f6    _ZN13GCTaskManager8get_taskEj + 0x66
0x0632843b    _ZN12GCTaskThread3runEv + 0xdb
0x06506929    _Z10java_startP6Thread + 0x139
0xf76dcf50    start_thread + 0xa0
----------------- 22546 -----------------
0xf76e08d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071    _ZN7Monitor4waitEblb + 0x221
0x063272f6    _ZN13GCTaskManager8get_taskEj + 0x66
0x0632843b    _ZN12GCTaskThread3runEv + 0xdb
0x06506929    _Z10java_startP6Thread + 0x139
0xf76dcf50    start_thread + 0xa0
----------------- 22549 -----------------
0xf76e0be8    __pthread_cond_timedwait + 0xe8
0x340707e0    ????????
sun.jvm.hotspot.debugger.UnmappedAddressException
    at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
    at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
    at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
    at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
    at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 22550 -----------------
0xf76e08d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x065a1a31    _ZN13ObjectMonitor4waitExbP6Thread + 0x571
0x0659f3a2    _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
0x063c834a    JVM_MonitorWait + 0x1da
0xf3f944aa    * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xf3f8d08d    * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0xf3f8d08d    * java.lang.ref.Reference$ReferenceHandler.run() bci:46
line:116 (Interpreted frame)
0xf3f8a34c    <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5    _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf76dcf50    start_thread + 0xa0
----------------- 22551 -----------------
0xf76e08d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x065a1a31    _ZN13ObjectMonitor4waitExbP6Thread + 0x571
0x0659f3a2    _ZN18ObjectSynchronizer4waitE6HandlexP6Thread + 0x42
0x063c834a    JVM_MonitorWait + 0x1da
0xf3f944aa    * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xf3f8d08d    * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:116
(Interpreted frame)
0xf3f8cf27    * java.lang.ref.ReferenceQueue.remove() bci:2 line:132
(Interpreted frame)
0xf3f8cf27    * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159
(Interpreted frame)
0xf3f8a34c    <StubRoutines>
0x06375c9d
_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread
+ 0x15d
0x065057f8
_ZN2os20os_exception_wrapperEPFvP9JavaValueP12methodHandleP17JavaCallArgumentsP6ThreadES1_S3_S5_S7_
+ 0x18
0x063755b0
_ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandle12symbolHandleS3_P17JavaCallArgumentsP6Thread
+ 0xa0
0x0637563d
_ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandle12symbolHandleS4_P6Thread
+ 0x5d
0x063e5ca5    _Z12thread_entryP10JavaThreadP6Thread + 0x75
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf76dcf50    start_thread + 0xa0
----------------- 22554 -----------------
0xf76e28b0    __new_sem_wait + 0x40
0x06503c0d    _ZN2os11signal_waitEv + 0xd
0x06500d28    _Z19signal_thread_entryP10JavaThreadP6Thread + 0x28
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf76dcf50    start_thread + 0xa0
----------------- 22555 -----------------
0xf76e08d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064edf21    _ZN7Monitor4waitEblb + 0xd1
0x062ada2f    _ZN12CompileQueue3getEv + 0x5f
0x062af2b4    _ZN13CompileBroker20compiler_thread_loopEv + 0x184
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf76dcf50    start_thread + 0xa0
----------------- 22556 -----------------
0xf76e08d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064edf21    _ZN7Monitor4waitEblb + 0xd1
0x062ada2f    _ZN12CompileQueue3getEv + 0x5f
0x062af2b4    _ZN13CompileBroker20compiler_thread_loopEv + 0x184
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf76dcf50    start_thread + 0xa0
----------------- 22557 -----------------
0xf76e08d0    __pthread_cond_wait + 0x90
0x065060c9    _ZN2os13PlatformEvent4parkEv + 0xb9
0x064ed92d    _ZN7Monitor5IWaitEP6Threadx + 0xcd
0x064ee071    _ZN7Monitor4waitEblb + 0x221
0x064bc935
_ZN17LowMemoryDetector32low_memory_detector_thread_entryEP10JavaThreadP6Thread
+ 0x85
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf76dcf50    start_thread + 0xa0
----------------- 22558 -----------------
0xf76e0be8    __pthread_cond_timedwait + 0xe8
0x38ed9008    ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
address: 0x4bbbb06f with alignment: 4
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
    at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
    at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
    at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 22644 -----------------
0xf76e0be8    __pthread_cond_timedwait + 0xe8
0x07b9bcc8    ????????
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at
address: 0x4bbbb087 with alignment: 4
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:469)
    at
sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
    at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
    at
sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
    at
sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
    at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 24136 -----------------
0xf76e36fe    __libc_accept + 0x3e
0x061c1a78    _ZN14AttachListener7dequeueEv + 0x68
0x061c0e55    _Z28attach_listener_thread_entryP10JavaThreadP6Thread + 0x35
0x065cde5d    _ZN10JavaThread3runEv + 0xfd
0x06506929    _Z10java_startP6Thread + 0x139
0xf76dcf50    start_thread + 0xa0
----------------- 22510 -----------------
0xf76dd452    pthread_join + 0x112
0x0804dd78    ContinueInNewThread + 0x88
0x080497f6    main + 0x276
0xf759d775    ????????


[erlf07@cohiba ~ ]$ cat thread_dump_dn
Attaching to process ID 3836, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.2-b01
Deadlock Detection:

No deadlocks found.

Thread 7801: (state = BLOCKED)


Thread 4062: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275, line=565
(Interpreted frame)


Thread 4060: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275, line=565
(Interpreted frame)


Thread 4055: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - org.apache.hadoop.fs.DU$DURefreshThread.run() @bci=17, line=80
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)


Thread 3878: (state = IN_NATIVE)
 - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0
(Interpreted frame)
 - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=384
(Interpreted frame)
 - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453
(Interpreted frame)
 - java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame)
 - sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() @bci=1,
line=34 (Interpreted frame)
 - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop()
@bci=55, line=369 (Interpreted frame)
 - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=341
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)


Thread 3872: (state = BLOCKED)


Thread 3871: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted
frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
(Interpreted frame)


Thread 3870: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
(Interpreted frame)


Thread 3864: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Interpreted
frame)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
(Interpreted frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
(Interpreted frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
(Interpreted frame)
 - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
(Interpreted frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
(Interpreted frame)
 - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
@bci=12, line=453 (Interpreted frame)
 - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
line=123 (Interpreted frame)
 - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4, line=118
(Interpreted frame)
 - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
line=114 (Interpreted frame)
 - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
line=171 (Interpreted frame)
 - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
(Interpreted frame)
 - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
frame)
 - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
 - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
line=139 (Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
line=168 (Interpreted frame)
 - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12, line=67
(Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
(Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
(Interpreted frame)
 - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1, line=537
(Interpreted frame)
 - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
(Interpreted frame)
 - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123, line=1234
(Interpreted frame)
 - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
(Interpreted frame)
 - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
(Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32, line=152
(Interpreted frame)
 - org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
line=156 (Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
(Interpreted frame)
 - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461 (Interpreted
frame)
 -
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration,
java.util.AbstractList) @bci=916, line=375 (Interpreted frame)
 -
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(org.apache.hadoop.conf.Configuration,
java.util.AbstractList) @bci=158, line=216 (Interpreted frame)
 -
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(java.lang.String[],
org.apache.hadoop.conf.Configuration) @bci=96, line=1283 (Interpreted frame)
 -
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(java.lang.String[],
org.apache.hadoop.conf.Configuration) @bci=92, line=1238 (Interpreted frame)
 -
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(java.lang.String[],
org.apache.hadoop.conf.Configuration) @bci=2, line=1246 (Interpreted frame)
 - org.apache.hadoop.hdfs.server.datanode.DataNode.main(java.lang.String[])
@bci=12, line=1368 (Interpreted frame)




[erlf07@cohiba ~ ]$ cat thread_dump_tt
Attaching to process ID 22510, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.2-b01
Deadlock Detection:

No deadlocks found.

Thread 24136: (state = BLOCKED)


Thread 22644: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=275, line=565
(Interpreted frame)


Thread 22554: (state = BLOCKED)


Thread 22551: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted
frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted
frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
(Interpreted frame)


Thread 22550: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
(Interpreted frame)


Thread 22544: (state = IN_NATIVE)
 - java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Interpreted
frame)
 - java.io.FileInputStream.read(byte[], int, int) @bci=4, line=199
(Interpreted frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=39, line=256
(Interpreted frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
(Interpreted frame)
 - java.io.BufferedInputStream.fill() @bci=175, line=218 (Interpreted frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=258
(Interpreted frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=317
(Interpreted frame)
 - sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte()
@bci=12, line=453 (Interpreted frame)
 - sun.security.provider.SeedGenerator.getSeedBytes(byte[]) @bci=11,
line=123 (Interpreted frame)
 - sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4, line=118
(Interpreted frame)
 - sun.security.provider.SecureRandom.engineGenerateSeed(int) @bci=5,
line=114 (Interpreted frame)
 - sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=40,
line=171 (Interpreted frame)
 - java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=433
(Interpreted frame)
 - java.security.SecureRandom.next(int) @bci=17, line=455 (Interpreted
frame)
 - java.util.Random.nextLong() @bci=3, line=284 (Interpreted frame)
 - org.mortbay.jetty.servlet.HashSessionIdManager.doStart() @bci=73,
line=139 (Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.servlet.AbstractSessionManager.doStart() @bci=96,
line=168 (Interpreted frame)
 - org.mortbay.jetty.servlet.HashSessionManager.doStart() @bci=12, line=67
(Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.servlet.SessionHandler.doStart() @bci=4, line=115
(Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
(Interpreted frame)
 - org.mortbay.jetty.handler.ContextHandler.startContext() @bci=1, line=537
(Interpreted frame)
 - org.mortbay.jetty.servlet.Context.startContext() @bci=1, line=136
(Interpreted frame)
 - org.mortbay.jetty.webapp.WebAppContext.startContext() @bci=123, line=1234
(Interpreted frame)
 - org.mortbay.jetty.handler.ContextHandler.doStart() @bci=140, line=517
(Interpreted frame)
 - org.mortbay.jetty.webapp.WebAppContext.doStart() @bci=170, line=460
(Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.handler.HandlerCollection.doStart() @bci=32, line=152
(Interpreted frame)
 - org.mortbay.jetty.handler.ContextHandlerCollection.doStart() @bci=5,
line=156 (Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.mortbay.jetty.handler.HandlerWrapper.doStart() @bci=11, line=130
(Interpreted frame)
 - org.mortbay.jetty.Server.doStart() @bci=201, line=222 (Interpreted frame)
 - org.mortbay.component.AbstractLifeCycle.start() @bci=31, line=50
(Interpreted frame)
 - org.apache.hadoop.http.HttpServer.start() @bci=383, line=461 (Interpreted
frame)
 -
org.apache.hadoop.mapred.TaskTracker.<init>(org.apache.hadoop.mapred.JobConf)
@bci=519, line=931 (Interpreted frame)
 - org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) @bci=52,
line=2833 (Interpreted frame)









Edson Ramiro


On 6 April 2010 19:47, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Edson,
>
> Your attachments did not come through - can you put them on pastebin?
>
> -Todd
>
> On Tue, Apr 6, 2010 at 3:37 PM, Edson Ramiro <erlfilho@gmail.com> wrote:
> > Hi Todd,
> >
> > I'm getting this behavior in another cluster too, there the same thing
> > happens.
> > and as I don't have a jstack installed in the first cluster and
> > I'm not the admin, I'm sending the results of the second cluster.
> >
> > These are the results:
> >
> > [erlf07@cohiba ~ ]$ jstack  -l 22510
> > 22510: well-known file is not secure
> > [erlf07@cohiba ~ ]$ jstack -l 3836
> > 3836: well-known file is not secure
> >
> > The jstack -F result is in thread_dump files and the jstack -m result is
> in
> > java_native_frames.
> >
> > The files ending with nn are the namenode results and the files ending
> with
> > dn are the datanode results.
> >
> > Thanks,
> >
> > Edson Ramiro
> >
> >
> > On 6 April 2010 18:19, Todd Lipcon <todd@cloudera.com> wrote:
> >>
> >> Hi Edson,
> >>
> >> Can you please run "jstack" on the daemons in question and paste the
> >> output here?
> >>
> >> -Todd
> >>
> >> On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro <erlfilho@gmail.com>
> wrote:
> >> > Hi all,
> >> >
> >> > I configured the Hadoop in a cluster and the NameNode and JobTracker
> are
> >> > running ok, but the DataNode and TaskTracker Doesn't start, they stop
> >> > and
> >> > keep waiting
> >> > when they are going to start the Jetty
> >> >
> >> > I observed that Jetty can't start the _SelectChannelConnector_
> >> >
> >> > Is there any Jetty configuration that should be changed ?
> >> >
> >> > There is no log message in the NN and JT when I try to start the DN
> and
> >> > TT.
> >> >
> >> > The kernel I'm using is:
> >> > Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64
> GNU/Linux
> >> >
> >> > This is the message when I start the DN. It happens with TT too.
> >> >
> >> > ramiro@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode
> >> > 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG:
> >> > /************************************************************
> >> > STARTUP_MSG: Starting DataNode
> >> > STARTUP_MSG:   host = bl05.ctinfra.ufpr.br/192.168.1.115
> >> > STARTUP_MSG:   args = []
> >> > STARTUP_MSG:   version = 0.20.1+169.56
> >> > STARTUP_MSG:   build =  -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3;
> >> > compiled by 'chad' on Tue Feb  2 13:27:17 PST 2010
> >> > ************************************************************/
> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Registered
> >> > FSDatasetStatusMBean
> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010
> >> > 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is
> 1048576
> >> > bytes/s
> >> > 10/04/06 16:24:14 INFO mortbay.log: Logging to
> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> >> > org.mortbay.log.Slf4jLog
> >> > 10/04/06 16:24:14 INFO http.HttpServer: Port returned by
> >> > webServer.getConnectors()[0].getLocalPort() before open() is -1.
> Opening
> >> > the
> >> > listener on 50075
> >> > 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort()
> returned
> >> > 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
> >> > 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075
> >> > 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14
> >> >
> >> > Thanks in Advance,
> >> >
> >> > Edson Ramiro
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message