hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yaoxiaohua <yaoxiao...@outlook.com>
Subject nodemanager process crash for array index out of bound
Date Fri, 18 Dec 2015 03:02:30 GMT
Hi guys,

Nodemanager process error log:

Could you give me some tips for this , why happened?

 

2015-12-16 17:24:38,605 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Container
sMonitorImpl: Uncaught exception in ContainerMemoryManager while managing
memory of container_1448915696877_23390_01_000430

java.lang.ArrayIndexOutOfBoundsException

        at
sun.misc.FormattedFloatingDecimal.dtoa(FormattedFloatingDecimal.java:848)

        at
sun.misc.FormattedFloatingDecimal.<init>(FormattedFloatingDecimal.java:542)

        at java.util.Formatter$FormatSpecifier.print(Formatter.java:3264)

        at java.util.Formatter$FormatSpecifier.print(Formatter.java:3202)

        at
java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2769)

        at java.util.Formatter$FormatSpecifier.print(Formatter.java:2720)

        at java.util.Formatter.format(Formatter.java:2500)

        at java.util.Formatter.format(Formatter.java:2435)

        at java.lang.String.format(String.java:2148)

        at org.apache.hadoop.util.StringUtils.format(StringUtils.java:123)

        at
org.apache.hadoop.util.StringUtils$TraditionalBinaryPrefix.long2String(Strin
gUtils.java:758)

        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Container
sMonitorImpl$MonitoringThread.formatUsageString(ContainersMonitorImpl.java:4
87)

        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Container
sMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:399)

2015-12-16 17:24:39,663 ERROR org.apache.hadoop.mapred.ShuffleHandler:
Shuffle error: 

java.lang.ArrayIndexOutOfBoundsException

        at sun.nio.cs.UTF_8$Encoder.encode(UTF_8.java:652)

        at java.lang.StringCoding.encode(StringCoding.java:899)

        at java.lang.String.getBytes(String.java:2221)

        at
org.apache.hadoop.mapreduce.security.SecureShuffleUtils.verifyReply(SecureSh
uffleUtils.java:100)

        at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.verifyRequest(ShuffleHandler
.java:558)

        at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandl
er.java:499)

        at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWri
teHandler.java:142)

        at
org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpC
hunkAggregator.java:148)

        at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)

        at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceive
d(FrameDecoder.java:459)

        at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDe
coder.java:536)

        at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Replay
ingDecoder.java:435)

        at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)

        at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)

        at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

        at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:107)

        at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)

        at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:88)

        at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
57)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
27)

        at java.lang.Thread.run(Thread.java:809)

2015-12-16 17:24:39,663 ERROR org.apache.hadoop.mapred.ShuffleHandler:
Shuffle error [id: 0x34ed4c97, /172.19.206.148:34641 =>
/172.19.206.142:8080] EXCEPTION: java.lang.ArrayIndexOutOfBoundsException

2015-12-16 17:24:39,663 FATAL
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Container
Monitor,5,main] threw an Error.  Shutting down now...

java.lang.OutOfMemoryError: Java 堆空间(heapspace)

        at java.util.HashMap.inflateTable(HashMap.java:328)

        at java.util.HashMap.<init>(HashMap.java:308)

        at
org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.updateProcessTree(ProcfsB
asedProcessTree.java:154)

        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Container
sMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:390)

2015-12-16 17:24:39,666 INFO org.apache.hadoop.util.ExitUtil: Halt with
status -1 Message: HaltException

 

 

Best Regards,

Evan Yao


Mime
View raw message