Hi,
We built up a Hadoop cluster as one master node and 4 several slave nodes, using Hadoop 0.20.2
+ Cygwin on Windows 7. We submit job with Hadoop streaming. The mapper is a console application
written by C++, and no reducer. When streaming job is executed, sometime there is one task
will be in "initializing" state for quite long time (even one hour). After we increase slave
node to 6, this issue happened more frequently.
We checked jobtracker, tasktracker logs, there is no error/warning recorded.
On taskmgr, we found the java process hang. Below is the dump info. Any suggestion or solution?
Attaching to process ID 12832, please wait...
Client compiler detected.
JVM version is 21.0-b17
Deadlock Detection:
No deadlocks found.
Finding object size using Printezis bits and skipping over...
Thread 14: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:178)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Locked ownable synchronizers:
- None
Thread 13: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net.InetSocketAddress,
java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=49, line=726 (Interpreted
frame)
- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[])
@bci=48, line=220 (Interpreted frame)
- org.apache.hadoop.mapred.$Proxy0.ping(org.apache.hadoop.mapred.TaskAttemptID) @bci=16 (Interpreted
frame)
- org.apache.hadoop.mapred.Task$TaskReporter.run() @bci=168, line=548 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 11: (state = IN_NATIVE)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(long, int, int[], int[], int[], long) @bci=0
(Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll() @bci=43, line=295 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(sun.nio.ch.WindowsSelectorImpl$SubSelector)
@bci=1, line=277 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl.doSelect(long) @bci=63, line=158 (Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Interpreted frame)
- org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(java.nio.channels.SelectableChannel,
int, long) @bci=46, line=332 (Interpreted frame)
- org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) @bci=80, line=157
(Interpreted frame)
- org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, line=155 (Interpreted
frame)
- org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, line=128 (Interpreted
frame)
- java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) @bci=4, line=276
(Interpreted frame)
- java.io.BufferedInputStream.fill() @bci=175, line=235 (Interpreted frame)
- java.io.BufferedInputStream.read() @bci=12, line=254 (Interpreted frame)
- java.io.DataInputStream.readInt() @bci=4, line=387 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection.receiveResponse() @bci=19, line=501 (Interpreted
frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=446 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 10: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:178)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Locked ownable synchronizers:
- None
Thread 6: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 5: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 4: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Locked ownable synchronizers:
- None
Thread 3: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Locked ownable synchronizers:
- None
Thread 1: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net.InetSocketAddress,
java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=49, line=726 (Interpreted
frame)
- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[])
@bci=48, line=220 (Interpreted frame)
- org.apache.hadoop.mapred.$Proxy0.commitPending(org.apache.hadoop.mapred.TaskAttemptID, org.apache.hadoop.mapred.TaskStatus)
@bci=20 (Interpreted frame)
- org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter)
@bci=80, line=678 (Interpreted frame)
- org.apache.hadoop.mapred.MapTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol)
@bci=116, line=309 (Interpreted frame)
- org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=455, line=170 (Interpreted
frame)
Locked ownable synchronizers:
- None
|