hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2947) Sort fails on YARN+MR with lots of task failures
Date Thu, 08 Sep 2011 20:08:08 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100652#comment-13100652
] 

Hudson commented on MAPREDUCE-2947:
-----------------------------------

Integrated in Hadoop-Mapreduce-trunk #811 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/811/])
    MAPREDUCE-2947. Fixed race condition in AuxiliaryServices. Contributed by Vinod K V.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166849
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/StartContainerResponsePBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java


> Sort fails on YARN+MR with lots of task failures
> ------------------------------------------------
>
>                 Key: MAPREDUCE-2947
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2947
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2947-20110907.txt
>
>
> [~karams](the great man the world hardly knows about) found lots of failing tasks while
running sort on a 350 node cluster. The failed tasks eventually failed the job and this happening
consistently on the big cluster.
> {quote}
> Container launch failed for container_1315410418107_0002_01_002511 : RemoteTrace: java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:129)
at java.nio.ByteBuffer.get(ByteBuffer.java:675) at com.google.protobuf.ByteString.copyFrom(ByteString.java:108)
at com.google.protobuf.ByteString.copyFrom(ByteString.java:117) at org.apache.hadoop.yarn.util.ProtoUtils.convertToProtoFormat(ProtoUtils.java:97)
at org.apache.hadoop.yarn.api.records.ProtoBase.convertToProtoFormat(ProtoBase.java:59) at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.access$100(StartContainerResponsePBImpl.java:35)
at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl$1$1.next(StartContainerResponsePBImpl.java:134)
at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl$1$1.next(StartContainerResponsePBImpl.java:122)
at com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:319) at
org.apache.hadoop.yarn.proto.YarnServiceProtos$StartContainerResponseProto$Builder.addAllServiceResponse(YarnServiceProtos.java:12620)
at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.addServiceResponseToProto(StartContainerResponsePBImpl.java:144)
at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.mergeLocalToBuilder(StartContainerResponsePBImpl.java:60)
at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.mergeLocalToProto(StartContainerResponsePBImpl.java:68)
at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerResponsePBImpl.getProto(StartContainerResponsePBImpl.java:52)
at org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagerPBServiceImpl.startContainer(ContainerManagerPBServiceImpl.java:69)
at org.apache.hadoop.yarn.proto.ContainerManager$ContainerManagerService$2.callBlockingMethod(ContainerManager.java:83)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:337)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1496) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1492)
at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1490) at LocalTrace: org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:151)
at $Proxy20.startContainer(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:81)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:215)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message