Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3E7419924 for ; Mon, 10 Sep 2012 13:32:03 +0000 (UTC) Received: (qmail 82118 invoked by uid 500); 10 Sep 2012 13:31:56 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 82040 invoked by uid 500); 10 Sep 2012 13:31:56 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Delivered-To: moderator for user@hadoop.apache.org Received: (qmail 33546 invoked by uid 99); 10 Sep 2012 08:46:17 -0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.80,396,1344236400"; d="scan'208,217";a="191053218" From: "Wu, Mandy" To: "user@hadoop.apache.org" CC: "Wu, Mandy" Subject: Hadoop long initializing issue Thread-Topic: Hadoop long initializing issue Thread-Index: Ac2PLEBuYWDVxRTHTpWSz79ah326Zw== Importance: high X-Priority: 1 Date: Mon, 10 Sep 2012 08:45:38 +0000 Message-ID: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: multipart/alternative; boundary="_000_B858914AA9F2174489253548397BDA670FD82E59SHSMSX102ccrcor_" MIME-Version: 1.0 --_000_B858914AA9F2174489253548397BDA670FD82E59SHSMSX102ccrcor_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, We built up a Hadoop cluster as one master node and 4 several slave nodes, = using Hadoop 0.20.2 + Cygwin on Windows 7. We submit job with Hadoop stream= ing. The mapper is a console application written by C++, and no reducer. Wh= en streaming job is executed, sometime there is one task will be in "initia= lizing" state for quite long time (even one hour). After we increase slave = node to 6, this issue happened more frequently. We checked jobtracker, tasktracker logs, there is no error/warning recorded= . On taskmgr, we found the java process hang. Below is the dump info. Any sug= gestion or solution? Attaching to process ID 12832, please wait... Client compiler detected. JVM version is 21.0-b17 Deadlock Detection: No deadlocks found. Finding object size using Printezis bits and skipping over... Thread 14: (state =3D BLOCKED) - java.lang.Thread.sleep(long) @bci=3D0 (Compiled frame; information may be= imprecise) Error occurred during stack walking: java.lang.NullPointerException at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224) at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame= .java:342) at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283) at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194) at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199) at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119) at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:17= 8) at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl= .java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) at sun.tools.jstack.JStack.main(JStack.java:102) Locked ownable synchronizers: - None Thread 13: (state =3D BLOCKED) - java.lang.Object.wait(long) @bci=3D0 (Interpreted frame) - java.lang.Object.wait() @bci=3D2, line=3D503 (Interpreted frame) - org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net= .InetSocketAddress, java.lang.Class, org.apache.hadoop.security.UserGroupIn= formation) @bci=3D49, line=3D726 (Interpreted frame) - org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.refl= ect.Method, java.lang.Object[]) @bci=3D48, line=3D220 (Interpreted frame) - org.apache.hadoop.mapred.$Proxy0.ping(org.apache.hadoop.mapred.TaskAttemp= tID) @bci=3D16 (Interpreted frame) - org.apache.hadoop.mapred.Task$TaskReporter.run() @bci=3D168, line=3D548 (= Interpreted frame) - java.lang.Thread.run() @bci=3D11, line=3D722 (Interpreted frame) Locked ownable synchronizers: - None Thread 11: (state =3D IN_NATIVE) - sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(long, int, int[], int[],= int[], long) @bci=3D0 (Interpreted frame) - sun.nio.ch.WindowsSelectorImpl$SubSelector.poll() @bci=3D43, line=3D295 (= Interpreted frame) - sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(sun.nio.ch.WindowsS= electorImpl$SubSelector) @bci=3D1, line=3D277 (Interpreted frame) - sun.nio.ch.WindowsSelectorImpl.doSelect(long) @bci=3D63, line=3D158 (Inte= rpreted frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=3D37, line=3D87 (Inter= preted frame) - sun.nio.ch.SelectorImpl.select(long) @bci=3D30, line=3D98 (Interpreted fr= ame) - org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(java.nio.ch= annels.SelectableChannel, int, long) @bci=3D46, line=3D332 (Interpreted fra= me) - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) = @bci=3D80, line=3D157 (Interpreted frame) - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=3D= 6, line=3D155 (Interpreted frame) - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=3D7, = line=3D128 (Interpreted frame) - java.io.FilterInputStream.read(byte[], int, int) @bci=3D7, line=3D133 (In= terpreted frame) - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int,= int) @bci=3D4, line=3D276 (Interpreted frame) - java.io.BufferedInputStream.fill() @bci=3D175, line=3D235 (Interpreted fr= ame) - java.io.BufferedInputStream.read() @bci=3D12, line=3D254 (Interpreted fra= me) - java.io.DataInputStream.readInt() @bci=3D4, line=3D387 (Interpreted frame= ) - org.apache.hadoop.ipc.Client$Connection.receiveResponse() @bci=3D19, line= =3D501 (Interpreted frame) - org.apache.hadoop.ipc.Client$Connection.run() @bci=3D62, line=3D446 (Inte= rpreted frame) Locked ownable synchronizers: - None Thread 10: (state =3D BLOCKED) - java.lang.Thread.sleep(long) @bci=3D0 (Compiled frame; information may be= imprecise) Error occurred during stack walking: java.lang.NullPointerException at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224) at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame= .java:342) at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283) at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194) at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199) at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119) at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:17= 8) at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl= .java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) at sun.tools.jstack.JStack.main(JStack.java:102) Locked ownable synchronizers: - None Thread 6: (state =3D BLOCKED) Locked ownable synchronizers: - None Thread 5: (state =3D BLOCKED) Locked ownable synchronizers: - None Thread 4: (state =3D BLOCKED) - java.lang.Object.wait(long) @bci=3D0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=3D44, line=3D135 (Compiled= frame) Error occurred during stack walking: java.lang.NullPointerException at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224) at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404) at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289) at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194) at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199) at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119) at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl= .java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) at sun.tools.jstack.JStack.main(JStack.java:102) Locked ownable synchronizers: - None Thread 3: (state =3D BLOCKED) - java.lang.Object.wait(long) @bci=3D0 (Interpreted frame) - java.lang.Object.wait() @bci=3D2, line=3D503 (Compiled frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=3D46, line=3D133 (Com= piled frame) Error occurred during stack walking: java.lang.NullPointerException at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224) at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404) at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289) at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194) at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199) at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119) at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl= .java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) at sun.tools.jstack.JStack.main(JStack.java:102) Locked ownable synchronizers: - None Thread 1: (state =3D BLOCKED) - java.lang.Object.wait(long) @bci=3D0 (Interpreted frame) - java.lang.Object.wait() @bci=3D2, line=3D503 (Interpreted frame) - org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net= .InetSocketAddress, java.lang.Class, org.apache.hadoop.security.UserGroupIn= formation) @bci=3D49, line=3D726 (Interpreted frame) - org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.refl= ect.Method, java.lang.Object[]) @bci=3D48, line=3D220 (Interpreted frame) - org.apache.hadoop.mapred.$Proxy0.commitPending(org.apache.hadoop.mapred.T= askAttemptID, org.apache.hadoop.mapred.TaskStatus) @bci=3D20 (Interpreted f= rame) - org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilical= Protocol, org.apache.hadoop.mapred.Task$TaskReporter) @bci=3D80, line=3D678= (Interpreted frame) - org.apache.hadoop.mapred.MapTask.run(org.apache.hadoop.mapred.JobConf, or= g.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=3D116, line=3D309 (Inter= preted frame) - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=3D455, line= =3D170 (Interpreted frame) Locked ownable synchronizers: - None --_000_B858914AA9F2174489253548397BDA670FD82E59SHSMSX102ccrcor_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

We built up a Hadoop cluster as= one master node and 4 several slave nodes, using Hadoop 0.20.2 + Cygwi= n on Windows 7. We submit job with Hadoop streaming. The mapper is a consol= e application written by C++, and no reducer. When streaming job is executed, sometime there is one task will be in R= 20;initializing” state for quite long time (even one hour). After we = increase slave node to 6, this issue happened more frequently.

 

We checked jobtracker, tasktrac= ker logs, there is no error/warning recorded.

 

On taskmgr, we found the java p= rocess hang. Below is the dump info. Any suggestion or solution?=

 

Attaching to process ID 12832, please wait...

Client compiler detected.

JVM version is 21.0-b17

Deadlock Detection:

 

No deadlocks found.

 

Finding object size using Printezis bits and skipping = over...

Thread 14: (state =3D BLOCKED)

- java.lang.Thread.sleep(long) @bci=3D0 (Compiled fram= e; information may be imprecise)

Error occurred during stack walking:=

java.lang.NullPointerException

    at sun.jvm.hotspot.runtime.Frame.ad= dressOfStackSlot(Frame.java:224)

    at sun.jvm.hotspot.runtime.x86.X86F= rame.senderForCompiledFrame(X86Frame.java:342)

    at sun.jvm.hotspot.runtime.x86.X86F= rame.sender(X86Frame.java:283)

    at sun.jvm.hotspot.runtime.Frame.se= nder(Frame.java:194)

    at sun.jvm.hotspot.runtime.Frame.re= alSender(Frame.java:199)

    at sun.jvm.hotspot.runtime.VFrame.s= ender(VFrame.java:119)

    at sun.jvm.hotspot.runtime.Compiled= VFrame.sender(CompiledVFrame.java:178)

    at sun.jvm.hotspot.runtime.VFrame.j= avaSender(VFrame.java:146)

    at sun.jvm.hotspot.tools.StackTrace= .run(StackTrace.java:76)

    at sun.jvm.hotspot.tools.StackTrace= .run(StackTrace.java:45)

    at sun.jvm.hotspot.tools.JStack.run= (JStack.java:60)

    at sun.jvm.hotspot.tools.Tool.start= (Tool.java:221)

    at sun.jvm.hotspot.tools.JStack.mai= n(JStack.java:86)

    at sun.reflect.NativeMethodAccessor= Impl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessor= Impl.invoke(NativeMethodAccessorImpl.java:57)

    at sun.reflect.DelegatingMethodAcce= ssorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(= Method.java:601)

    at sun.tools.jstack.JStack.runJStac= kTool(JStack.java:136)

    at sun.tools.jstack.JStack.main(JSt= ack.java:102)

 

Locked ownable synchronizers:

    - None

 

Thread 13: (state =3D BLOCKED)

- java.lang.Object.wait(long) @bci=3D0 (Interpreted fr= ame)

- java.lang.Object.wait() @bci=3D2, line=3D503 (Interp= reted frame)

- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.= io.Writable, java.net.InetSocketAddress, java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=3D49, line=3D726 (In= terpreted frame)

- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.O= bject, java.lang.reflect.Method, java.lang.Object[]) @bci=3D48, line=3D220 (Interpreted frame)

- org.apache.hadoop.mapred.$Proxy0.ping(org.apache.had= oop.mapred.TaskAttemptID) @bci=3D16 (Interpreted frame)

- org.apache.hadoop.mapred.Task$TaskReporter.run() @bc= i=3D168, line=3D548 (Interpreted frame)

- java.lang.Thread.run() @bci=3D11, line=3D722 (Interp= reted frame)

 

Locked ownable synchronizers:

    - None

 

Thread 11: (state =3D IN_NATIVE)

- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(lon= g, int, int[], int[], int[], long) @bci=3D0 (Interpreted frame)

- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll() @b= ci=3D43, line=3D295 (Interpreted frame)

- sun.nio.ch.WindowsSelectorImpl$SubSelector.access$40= 0(sun.nio.ch.WindowsSelectorImpl$SubSelector) @bci=3D1, line=3D277 (Interpreted frame)

- sun.nio.ch.WindowsSelectorImpl.doSelect(long) @bci= =3D63, line=3D158 (Interpreted frame)

- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci= =3D37, line=3D87 (Interpreted frame)

- sun.nio.ch.SelectorImpl.select(long) @bci=3D30, line= =3D98 (Interpreted frame)

- org.apache.hadoop.net.SocketIOWithTimeout$SelectorPo= ol.select(java.nio.channels.SelectableChannel, int, long) @bci=3D46, line=3D332 (Interpreted frame)

- org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.= nio.ByteBuffer, int) @bci=3D80, line=3D157 (Interpreted frame)

- org.apache.hadoop.net.SocketInputStream.read(java.ni= o.ByteBuffer) @bci=3D6, line=3D155 (Interpreted frame)

- org.apache.hadoop.net.SocketInputStream.read(byte[],= int, int) @bci=3D7, line=3D128 (Interpreted frame)

- java.io.FilterInputStream.read(byte[], int, int) @bc= i=3D7, line=3D133 (Interpreted frame)

- org.apache.hadoop.ipc.Client$Connection$PingInputStr= eam.read(byte[], int, int) @bci=3D4, line=3D276 (Interpreted frame)

- java.io.BufferedInputStream.fill() @bci=3D175, line= =3D235 (Interpreted frame)

- java.io.BufferedInputStream.read() @bci=3D12, line= =3D254 (Interpreted frame)

- java.io.DataInputStream.readInt() @bci=3D4, line=3D3= 87 (Interpreted frame)

- org.apache.hadoop.ipc.Client$Connection.receiveRespo= nse() @bci=3D19, line=3D501 (Interpreted frame)

- org.apache.hadoop.ipc.Client$Connection.run() @bci= =3D62, line=3D446 (Interpreted frame)

 

Locked ownable synchronizers:

    - None

 

Thread 10: (state =3D BLOCKED)

- java.lang.Thread.sleep(long) @bci=3D0 (Compiled fram= e; information may be imprecise)

Error occurred during stack walking:=

java.lang.NullPointerException

    at sun.jvm.hotspot.runtime.Frame.ad= dressOfStackSlot(Frame.java:224)

    at sun.jvm.hotspot.runtime.x86.X86F= rame.senderForCompiledFrame(X86Frame.java:342)

    at sun.jvm.hotspot.runtime.x86.X86F= rame.sender(X86Frame.java:283)

    at sun.jvm.hotspot.runtime.Frame.se= nder(Frame.java:194)

    at sun.jvm.hotspot.runtime.Frame.re= alSender(Frame.java:199)

    at sun.jvm.hotspot.runtime.VFrame.s= ender(VFrame.java:119)

    at sun.jvm.hotspot.runtime.Compiled= VFrame.sender(CompiledVFrame.java:178)

    at sun.jvm.hotspot.runtime.VFrame.j= avaSender(VFrame.java:146)

    at sun.jvm.hotspot.tools.StackTrace= .run(StackTrace.java:76)

    at sun.jvm.hotspot.tools.StackTrace= .run(StackTrace.java:45)

    at sun.jvm.hotspot.tools.JStack.run= (JStack.java:60)

    at sun.jvm.hotspot.tools.Tool.start= (Tool.java:221)

    at sun.jvm.hotspot.tools.JStack.mai= n(JStack.java:86)

    at sun.reflect.NativeMethodAccessor= Impl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessor= Impl.invoke(NativeMethodAccessorImpl.java:57)

    at sun.reflect.DelegatingMethodAcce= ssorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(= Method.java:601)

    at sun.tools.jstack.JStack.runJStac= kTool(JStack.java:136)

    at sun.tools.jstack.JStack.main(JSt= ack.java:102)

 

Locked ownable synchronizers:

    - None

 

Thread 6: (state =3D BLOCKED)

 

Locked ownable synchronizers:

    - None

 

Thread 5: (state =3D BLOCKED)

 

Locked ownable synchronizers:

    - None

 

Thread 4: (state =3D BLOCKED)

- java.lang.Object.wait(long) @bci=3D0 (Interpreted fr= ame)

- java.lang.ref.ReferenceQueue.remove(long) @bci=3D44,= line=3D135 (Compiled frame)

Error occurred during stack walking:=

java.lang.NullPointerException

    at sun.jvm.hotspot.runtime.Frame.ad= dressOfStackSlot(Frame.java:224)

    at sun.jvm.hotspot.runtime.x86.X86F= rame.getSenderSP(X86Frame.java:404)

    at sun.jvm.hotspot.runtime.x86.X86F= rame.sender(X86Frame.java:289)

    at sun.jvm.hotspot.runtime.Frame.se= nder(Frame.java:194)

    at sun.jvm.hotspot.runtime.Frame.re= alSender(Frame.java:199)

    at sun.jvm.hotspot.runtime.VFrame.s= ender(VFrame.java:119)

    at sun.jvm.hotspot.runtime.VFrame.j= avaSender(VFrame.java:151)

    at sun.jvm.hotspot.tools.StackTrace= .run(StackTrace.java:76)

    at sun.jvm.hotspot.tools.StackTrace= .run(StackTrace.java:45)

    at sun.jvm.hotspot.tools.JStack.run= (JStack.java:60)

    at sun.jvm.hotspot.tools.Tool.start= (Tool.java:221)

    at sun.jvm.hotspot.tools.JStack.mai= n(JStack.java:86)

    at sun.reflect.NativeMethodAccessor= Impl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessor= Impl.invoke(NativeMethodAccessorImpl.java:57)

    at sun.reflect.DelegatingMethodAcce= ssorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(= Method.java:601)

    at sun.tools.jstack.JStack.runJStac= kTool(JStack.java:136)

    at sun.tools.jstack.JStack.main(JSt= ack.java:102)

 

Locked ownable synchronizers:

    - None

 

Thread 3: (state =3D BLOCKED)

- java.lang.Object.wait(long) @bci=3D0 (Interpreted fr= ame)

- java.lang.Object.wait() @bci=3D2, line=3D503 (Compil= ed frame)

- java.lang.ref.Reference$ReferenceHandler.run() @bci= =3D46, line=3D133 (Compiled frame)

Error occurred during stack walking:=

java.lang.NullPointerException

    at sun.jvm.hotspot.runtime.Frame.ad= dressOfStackSlot(Frame.java:224)

    at sun.jvm.hotspot.runtime.x86.X86F= rame.getSenderSP(X86Frame.java:404)

    at sun.jvm.hotspot.runtime.x86.X86F= rame.sender(X86Frame.java:289)

    at sun.jvm.hotspot.runtime.Frame.se= nder(Frame.java:194)

    at sun.jvm.hotspot.runtime.Frame.re= alSender(Frame.java:199)

    at sun.jvm.hotspot.runtime.VFrame.s= ender(VFrame.java:119)

    at sun.jvm.hotspot.runtime.VFrame.j= avaSender(VFrame.java:151)

    at sun.jvm.hotspot.tools.StackTrace= .run(StackTrace.java:76)

    at sun.jvm.hotspot.tools.StackTrace= .run(StackTrace.java:45)

    at sun.jvm.hotspot.tools.JStack.run= (JStack.java:60)

    at sun.jvm.hotspot.tools.Tool.start= (Tool.java:221)

    at sun.jvm.hotspot.tools.JStack.mai= n(JStack.java:86)

    at sun.reflect.NativeMethodAccessor= Impl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessor= Impl.invoke(NativeMethodAccessorImpl.java:57)

    at sun.reflect.DelegatingMethodAcce= ssorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(= Method.java:601)

    at sun.tools.jstack.JStack.runJStac= kTool(JStack.java:136)

    at sun.tools.jstack.JStack.main(JSt= ack.java:102)

 

Locked ownable synchronizers:

    - None

 

Thread 1: (state =3D BLOCKED)

- java.lang.Object.wait(long) @bci=3D0 (Interpreted fr= ame)

- java.lang.Object.wait() @bci=3D2, line=3D503 (Interp= reted frame)

- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.= io.Writable, java.net.InetSocketAddress, java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=3D49, line=3D726 (In= terpreted frame)

- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.O= bject, java.lang.reflect.Method, java.lang.Object[]) @bci=3D48, line=3D220 (Interpreted frame)

- org.apache.hadoop.mapred.$Proxy0.commitPending(org.a= pache.hadoop.mapred.TaskAttemptID, org.apache.hadoop.mapred.TaskStatus) @bci=3D20 (Interpreted frame)

- org.apache.hadoop.mapred.Task.done(org.apache.hadoop= .mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter) @bci=3D80, line=3D678 (Interpreted frame)

- org.apache.hadoop.mapred.MapTask.run(org.apache.hado= op.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=3D116, line=3D309 (Interpreted frame)

- org.apache.hadoop.mapred.Child.main(java.lang.String= []) @bci=3D455, line=3D170 (Interpreted frame)

 

Locked ownable synchronizers:

    - None

 

 

 

 

--_000_B858914AA9F2174489253548397BDA670FD82E59SHSMSX102ccrcor_--