hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Child processes on datanodes/task trackers
Date Sat, 22 Dec 2012 14:01:45 GMT
This is an interesting stacktrace. The JVM had called exit but there
are threads stuck. Unsure if your JVM version (6u22) has anything to
do with it.

Are you using the JVM reuse feature?

On Sat, Dec 22, 2012 at 3:46 PM, Sedighe Tabatabaei
<tabatabaeised@gmail.com> wrote:
> Hello
> I wanted to know why I see running Child processes on my datanodes
> even though there is no job running at that time.
> when i use  jstack  <pid> command,result is:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x00007f2a74002000 nid=0x1f3
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "SIGTERM handler" daemon prio=10 tid=0x00007f2a74001000 nid=0x2fae
> waiting for monitor entry [0x00007f2a79891000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at java.lang.Shutdown.exit(Shutdown.java:168)
>         - waiting to lock <0x00007f2a9ffa68d0> (a java.lang.Class for
> java.lang.Shutdown)
>         at java.lang.Terminator$1.handle(Terminator.java:35)
>         at sun.misc.Signal$1.run(Signal.java:195)
>         at java.lang.Thread.run(Thread.java:662)
>
> "Thread-16" prio=10 tid=0x00007f2ae83d1000 nid=0x2f85 runnable
> [0x00007f2a9cb5a000]
>    java.lang.Thread.State: RUNNABLE
>         at java.lang.Throwable.fillInStackTrace(Native Method)
>         - locked <0x00007f2ad36e4d50> (a java.util.ConcurrentModificationException)
>         at java.lang.Throwable.<init>(Throwable.java:181)
>         at java.lang.Exception.<init>(Exception.java:29)
>         at java.lang.RuntimeException.<init>(RuntimeException.java:32)
>         at java.util.ConcurrentModificationException.<init>(ConcurrentModificationException.java:57)
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$KeyIterator.next(HashMap.java:828)
>         at com.sun.media.jai.codec.TempFileCleanupThread.run(FileCacheSeekableStream.java:315)
>
> "DestroyJavaVM" prio=10 tid=0x00007f2ae8009800 nid=0x1113 in
> Object.wait() [0x00007f2aedb81000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa5248528> (a
> com.sun.media.jai.codec.TempFileCleanupThread)
>         at java.lang.Thread.join(Thread.java:1186)
>         - locked <0x00007f2aa5248528> (a com.sun.media.jai.codec.TempFileCleanupThread)
>         at java.lang.Thread.join(Thread.java:1239)
>         at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
>         at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
>         at java.lang.Shutdown.runHooks(Shutdown.java:79)
>         at java.lang.Shutdown.sequence(Shutdown.java:123)
>         at java.lang.Shutdown.shutdown(Shutdown.java:190)
>         - locked <0x00007f2a9ffa68d0> (a java.lang.Class for java.lang.Shutdown)
>
> "Java2D Disposer" daemon prio=10 tid=0x00007f2ae8415000 nid=0x1775 in
> Object.wait() [0x00007f2a7a16d000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>         - locked <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>         at sun.java2d.Disposer.run(Disposer.java:127)
>         at java.lang.Thread.run(Thread.java:662)
>
> "Thread for syncLogs" daemon prio=10 tid=0x00007f2ae83bf000 nid=0x1208
> waiting on condition [0x00007f2a9f2fa000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007f2ae8093000 nid=0x112f
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00007f2ae8090800 nid=0x112d
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x00007f2ae808d800 nid=0x112a
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f2ae808b800 nid=0x1129
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f2ae806d000 nid=0x1126 in
> Object.wait() [0x00007f2a9fafd000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>         - locked <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00007f2ae806b000 nid=0x1124
> in Object.wait() [0x00007f2a9fbfe000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
>         at java.lang.Object.wait(Object.java:485)
>         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>         - locked <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00007f2ae8066800 nid=0x1122 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2ae801c800
> nid=0x1114 runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2ae801e800
> nid=0x1115 runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f2ae8020000
> nid=0x111c runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f2ae8022000
> nid=0x111f runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f2ae809d000 nid=0x1131
> waiting on condition
>
> JNI global references: 1774
>
>
> Thanks, Tabatabaei



--
Harsh J

Mime
View raw message