hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wu, Mandy" <mandy...@intel.com>
Subject Hadoop long initializing issue
Date Mon, 10 Sep 2012 08:45:38 GMT
Hi,
We built up a Hadoop cluster as one master node and 4 several slave nodes, using Hadoop 0.20.2
+ Cygwin on Windows 7. We submit job with Hadoop streaming. The mapper is a console application
written by C++, and no reducer. When streaming job is executed, sometime there is one task
will be in "initializing" state for quite long time (even one hour). After we increase slave
node to 6, this issue happened more frequently.

We checked jobtracker, tasktracker logs, there is no error/warning recorded.

On taskmgr, we found the java process hang. Below is the dump info. Any suggestion or solution?

Attaching to process ID 12832, please wait...
Client compiler detected.
JVM version is 21.0-b17
Deadlock Detection:

No deadlocks found.

Finding object size using Printezis bits and skipping over...
Thread 14: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
Error occurred during stack walking:
java.lang.NullPointerException
    at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
    at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
    at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
    at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
    at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
    at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
    at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:178)
    at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
    at sun.tools.jstack.JStack.main(JStack.java:102)

Locked ownable synchronizers:
    - None

Thread 13: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net.InetSocketAddress,
java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=49, line=726 (Interpreted
frame)
- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[])
@bci=48, line=220 (Interpreted frame)
- org.apache.hadoop.mapred.$Proxy0.ping(org.apache.hadoop.mapred.TaskAttemptID) @bci=16 (Interpreted
frame)
- org.apache.hadoop.mapred.Task$TaskReporter.run() @bci=168, line=548 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)

Locked ownable synchronizers:
    - None

Thread 11: (state = IN_NATIVE)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(long, int, int[], int[], int[], long) @bci=0
(Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.poll() @bci=43, line=295 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(sun.nio.ch.WindowsSelectorImpl$SubSelector)
@bci=1, line=277 (Interpreted frame)
- sun.nio.ch.WindowsSelectorImpl.doSelect(long) @bci=63, line=158 (Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Interpreted frame)
- org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(java.nio.channels.SelectableChannel,
int, long) @bci=46, line=332 (Interpreted frame)
- org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) @bci=80, line=157
(Interpreted frame)
- org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, line=155 (Interpreted
frame)
- org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, line=128 (Interpreted
frame)
- java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) @bci=4, line=276
(Interpreted frame)
- java.io.BufferedInputStream.fill() @bci=175, line=235 (Interpreted frame)
- java.io.BufferedInputStream.read() @bci=12, line=254 (Interpreted frame)
- java.io.DataInputStream.readInt() @bci=4, line=387 (Interpreted frame)
- org.apache.hadoop.ipc.Client$Connection.receiveResponse() @bci=19, line=501 (Interpreted
frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=446 (Interpreted frame)

Locked ownable synchronizers:
    - None

Thread 10: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
Error occurred during stack walking:
java.lang.NullPointerException
    at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
    at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
    at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
    at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
    at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
    at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
    at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:178)
    at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
    at sun.tools.jstack.JStack.main(JStack.java:102)

Locked ownable synchronizers:
    - None

Thread 6: (state = BLOCKED)

Locked ownable synchronizers:
    - None

Thread 5: (state = BLOCKED)

Locked ownable synchronizers:
    - None

Thread 4: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
    at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
    at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404)
    at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289)
    at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
    at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
    at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
    at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
    at sun.tools.jstack.JStack.main(JStack.java:102)

Locked ownable synchronizers:
    - None

Thread 3: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
    at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
    at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:404)
    at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:289)
    at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
    at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
    at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
    at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:151)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
    at sun.tools.jstack.JStack.main(JStack.java:102)

Locked ownable synchronizers:
    - None

Thread 1: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- org.apache.hadoop.ipc.Client.call(org.apache.hadoop.io.Writable, java.net.InetSocketAddress,
java.lang.Class, org.apache.hadoop.security.UserGroupInformation) @bci=49, line=726 (Interpreted
frame)
- org.apache.hadoop.ipc.RPC$Invoker.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[])
@bci=48, line=220 (Interpreted frame)
- org.apache.hadoop.mapred.$Proxy0.commitPending(org.apache.hadoop.mapred.TaskAttemptID, org.apache.hadoop.mapred.TaskStatus)
@bci=20 (Interpreted frame)
- org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter)
@bci=80, line=678 (Interpreted frame)
- org.apache.hadoop.mapred.MapTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol)
@bci=116, line=309 (Interpreted frame)
- org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=455, line=170 (Interpreted
frame)

Locked ownable synchronizers:
    - None





Mime
View raw message