hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mohammad Kamrul Islam (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
Date Fri, 24 Oct 2014 21:11:33 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183496#comment-14183496
] 

Mohammad Kamrul Islam commented on MAPREDUCE-6139:
--------------------------------------------------

I did some investigation/debugging and found the problem happened when ContainerLogAppender.close()
is iterating, the ContainerLogAppender.append() was executing by other thread. So append()
add an entry and then close() found the ConcurrentModificationException. 

The main question is : append() is a synchronized method and close() is within synchronized
block. The question is : how it is possible to execute both such synchronized methods at the
same time. Am I missing something?



> ConcurrentModificationException in ContainerLogAppender
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-6139
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Mohammad Kamrul Islam
>            Assignee: Mohammad Kamrul Islam
>            Priority: Critical
>
> Context:
> * Hadoop-2.3.0
> * Using Oozie 4.0.1
> * Pig version 0.11.x
> The job is submitted by Oozie to launch Pig script.
> The following exception traces were found on MR task log:
> In syslog:
> {noformat}
> 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook
'' failed, java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> 	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> 	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> 	at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> 	at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> 	at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> 	at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
> 	at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
> 	at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
> 	at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
> 	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Stopping MapTask metrics system...
> {noformat}
> in stderr:
> {noformat}
> java.util.ConcurrentModificationException
> 	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> 	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> 	at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> 	at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> 	at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> 	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
> 	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> 	at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> 	at org.apache.pig.Main.configureLog4J(Main.java:740)
> 	at org.apache.pig.Main.run(Main.java:384)
> 	at org.apache.pig.PigRunner.run(PigRunner.java:49)
> 	at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> 	at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> 	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> 	at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:483)
> 	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message