Return-Path: 
 <user-return-3791-apmail-hadoop-mapreduce-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4B9BBD415
	for <apmail-hadoop-mapreduce-user-archive@minotaur.apache.org>;
 Sat, 22 Dec 2012 14:02:39 +0000 (UTC)
Received: (qmail 88459 invoked by uid 500); 22 Dec 2012 14:02:34 -0000
Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org
Received: (qmail 88007 invoked by uid 500); 22 Dec 2012 14:02:34 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 87974 invoked by uid 99); 22 Dec 2012 14:02:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Dec 2012 14:02:33 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.223.177 as permitted sender)
Received: from [209.85.223.177] (HELO mail-ie0-f177.google.com)
 (209.85.223.177)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Dec 2012 14:02:26 +0000
Received: by mail-ie0-f177.google.com with SMTP id k13so7559569iea.36
        for <user@hadoop.apache.org>; Sat, 22 Dec 2012 06:02:06 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20120113;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:x-gm-message-state;
        bh=2sbMgliSrKlYUiOZt8kY/G7fa1B+Bfx0Eu7UkcMk27A=;
        b=ADFI+nk9mED9BG+2MuVk5/koUmoGT3xd+tB9zBXUewDLwXSWcEKqfNlIyTanzl9IS1
         ol292kRpyOSqD3V+pkW6stfZv08oqqeTHWc/9SMgGs6SZolVfG0UFl5N/g+yKITiOEU0
         B0+uu+9DYNNdnIjxO6qhYSIHOdwJi25bZ7ktp1iDfknhLVXtaFfd/cZfGm0teWCCZacV
         V20urMcHqzZw8aNQ59Q6aF4dcYDJA4EMesEWX24nG1Iit7vs4Xvn5Y+pSLtpU+hxutIA
         O1tScNQWe89Rv91x7q9qMUT8H5p1kdlGDSpsc2jDrKaaRMQwMVVV2PXB+yuWIKK9FqHe
         8Hkw==
Received: by 10.42.48.147 with SMTP id s19mr13946591icf.18.1356184925960; Sat,
 22 Dec 2012 06:02:05 -0800 (PST)
MIME-Version: 1.0
Received: by 10.64.6.129 with HTTP; Sat, 22 Dec 2012 06:01:45 -0800 (PST)
In-Reply-To: 
 <CAG7QZqgwcZm_A+sh=rTtyqBYT+-bxXPM+4ok3hWGq_O0n59CSg@mail.gmail.com>
References: 
 <CAG7QZqgwcZm_A+sh=rTtyqBYT+-bxXPM+4ok3hWGq_O0n59CSg@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Sat, 22 Dec 2012 19:31:45 +0530
Message-ID: 
 <CAOcnVr1qK_G0shUZg7H_MKrUh7yqOEMbU=+4Pfrbac+VQm_vsA@mail.gmail.com>
Subject: Re: Child processes on datanodes/task trackers
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
X-Gm-Message-State: 
 ALoCoQktDo/uXJJZgA/cQjVMC1XYQFvurYbJI+KBDERifxeRW8en2BhxzBN5Rn1A3rthMM+XDWZP
X-Virus-Checked: Checked by ClamAV on apache.org

This is an interesting stacktrace. The JVM had called exit but there
are threads stuck. Unsure if your JVM version (6u22) has anything to
do with it.

Are you using the JVM reuse feature?

On Sat, Dec 22, 2012 at 3:46 PM, Sedighe Tabatabaei
<tabatabaeised@gmail.com> wrote:
> Hello
> I wanted to know why I see running Child processes on my datanodes
> even though there is no job running at that time.
> when i use  jstack  <pid> command,result is:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x00007f2a74002000 nid=0x1f3
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "SIGTERM handler" daemon prio=10 tid=0x00007f2a74001000 nid=0x2fae
> waiting for monitor entry [0x00007f2a79891000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at java.lang.Shutdown.exit(Shutdown.java:168)
>         - waiting to lock <0x00007f2a9ffa68d0> (a java.lang.Class for
> java.lang.Shutdown)
>         at java.lang.Terminator$1.handle(Terminator.java:35)
>         at sun.misc.Signal$1.run(Signal.java:195)
>         at java.lang.Thread.run(Thread.java:662)
>
> "Thread-16" prio=10 tid=0x00007f2ae83d1000 nid=0x2f85 runnable
> [0x00007f2a9cb5a000]
>    java.lang.Thread.State: RUNNABLE
>         at java.lang.Throwable.fillInStackTrace(Native Method)
>         - locked <0x00007f2ad36e4d50> (a java.util.ConcurrentModificationException)
>         at java.lang.Throwable.<init>(Throwable.java:181)
>         at java.lang.Exception.<init>(Exception.java:29)
>         at java.lang.RuntimeException.<init>(RuntimeException.java:32)
>         at java.util.ConcurrentModificationException.<init>(ConcurrentModificationException.java:57)
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$KeyIterator.next(HashMap.java:828)
>         at com.sun.media.jai.codec.TempFileCleanupThread.run(FileCacheSeekableStream.java:315)
>
> "DestroyJavaVM" prio=10 tid=0x00007f2ae8009800 nid=0x1113 in
> Object.wait() [0x00007f2aedb81000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa5248528> (a
> com.sun.media.jai.codec.TempFileCleanupThread)
>         at java.lang.Thread.join(Thread.java:1186)
>         - locked <0x00007f2aa5248528> (a com.sun.media.jai.codec.TempFileCleanupThread)
>         at java.lang.Thread.join(Thread.java:1239)
>         at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
>         at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
>         at java.lang.Shutdown.runHooks(Shutdown.java:79)
>         at java.lang.Shutdown.sequence(Shutdown.java:123)
>         at java.lang.Shutdown.shutdown(Shutdown.java:190)
>         - locked <0x00007f2a9ffa68d0> (a java.lang.Class for java.lang.Shutdown)
>
> "Java2D Disposer" daemon prio=10 tid=0x00007f2ae8415000 nid=0x1775 in
> Object.wait() [0x00007f2a7a16d000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>         - locked <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>         at sun.java2d.Disposer.run(Disposer.java:127)
>         at java.lang.Thread.run(Thread.java:662)
>
> "Thread for syncLogs" daemon prio=10 tid=0x00007f2ae83bf000 nid=0x1208
> waiting on condition [0x00007f2a9f2fa000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007f2ae8093000 nid=0x112f
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00007f2ae8090800 nid=0x112d
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x00007f2ae808d800 nid=0x112a
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f2ae808b800 nid=0x1129
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f2ae806d000 nid=0x1126 in
> Object.wait() [0x00007f2a9fafd000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>         - locked <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00007f2ae806b000 nid=0x1124
> in Object.wait() [0x00007f2a9fbfe000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
>         at java.lang.Object.wait(Object.java:485)
>         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>         - locked <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00007f2ae8066800 nid=0x1122 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2ae801c800
> nid=0x1114 runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2ae801e800
> nid=0x1115 runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f2ae8020000
> nid=0x111c runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f2ae8022000
> nid=0x111f runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f2ae809d000 nid=0x1131
> waiting on condition
>
> JNI global references: 1774
>
>
> Thanks, Tabatabaei



--
Harsh J