hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
Date Tue, 28 Oct 2014 21:11:34 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187472#comment-14187472
] 

Sangjin Lee commented on MAPREDUCE-6139:
----------------------------------------

I think that call is there by design. The tail queue is an internal buffer and that code is
trying to write out the buffered log events. You'll also see super.close() right below this
for loop.

I think this is triggering the exception (not that it is the cause but that it triggers the
problem):

{noformat}
org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:197)
org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:118)
{noformat}

I haven't thought much about this, but perhaps another way to solve this is to modify append()
not to use the tail queue if it is in the middle of a shutdown.

> ConcurrentModificationException in ContainerLogAppender
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-6139
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Mohammad Kamrul Islam
>            Assignee: Mohammad Kamrul Islam
>            Priority: Critical
>         Attachments: ContainerLogAppender.java
>
>
> Context:
> * Hadoop-2.3.0
> * Using Oozie 4.0.1
> * Pig version 0.11.x
> The job is submitted by Oozie to launch Pig script.
> The following exception traces were found on MR task log:
> In syslog:
> {noformat}
> 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook
'' failed, java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> 	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> 	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> 	at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> 	at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> 	at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> 	at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
> 	at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
> 	at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
> 	at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
> 	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Stopping MapTask metrics system...
> {noformat}
> in stderr:
> {noformat}
> java.util.ConcurrentModificationException
> 	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> 	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> 	at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> 	at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> 	at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> 	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
> 	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> 	at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> 	at org.apache.pig.Main.configureLog4J(Main.java:740)
> 	at org.apache.pig.Main.run(Main.java:384)
> 	at org.apache.pig.PigRunner.run(PigRunner.java:49)
> 	at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> 	at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> 	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> 	at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:483)
> 	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message