Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@incubator.apache.org
Received-SPF: neutral (nike.apache.org: 216.139.236.26 is neither permitted
 nor denied by domain of abhijit.dhar.in@gmail.com)
Date: Fri, 22 Jun 2012 09:50:52 -0700 (PDT)
From: Abhijit Dhar <abhijit.dhar.in@gmail.com>
To: chukwa-user@hadoop.apache.org
Message-ID: <1340383852396-4025029.post@n3.nabble.com>
Subject: Deadlock happening in ChukwaDailyRollingFileAppender
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hi,

I am getting a deadlock in ChukwaDailyRollingFileAppender. The thread dump
is:

Found one Java-level deadlock:
=============================
"Thread-4":
  waiting to lock monitor 0x00007f1970004818 (object 0x00000000cb6669e8, a
java.lang.Class),
  which is held by "Thread for syncLogs"
"Thread for syncLogs":
  waiting to lock monitor 0x00007f1970004ea8 (object 0x00000000d65c44b8, a
org.apache.log4j.Logger),
  which is held by "main"
"main":
  waiting to lock monitor 0x00007f1970003d98 (object 0x00000000d65c3e80, a
org.apache.log4j.Logger),
  which is held by "communication thread"
"communication thread":
  waiting to lock monitor 0x0000000041eba250 (object 0x00000000d6736ec0, a
java.lang.Object),
  which is held by "main"

Java stack information for the threads listed above:
===================================================
"Thread-4":
        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:213)
        - waiting to lock <0x00000000cb6669e8> (a java.lang.Class for
org.apache.hadoop.mapred.TaskLog)
        at org.apache.hadoop.mapred.Child$1.run(Child.java:75)
"Thread for syncLogs":
        at org.apache.log4j.Category.getAllAppenders(Category.java:409)
        - waiting to lock <0x00000000d65c44b8> (a org.apache.log4j.Logger)
        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:218)
        - locked <0x00000000cb6669e8> (a java.lang.Class for
org.apache.hadoop.mapred.TaskLog)
        at org.apache.hadoop.mapred.Child$2.run(Child.java:89)
"main":
        at org.apache.log4j.Category.callAppenders(Category.java:202)
        - waiting to lock <0x00000000d65c3e80> (a org.apache.log4j.Logger)
        at org.apache.log4j.Category.forcedLog(Category.java:388)
        at org.apache.log4j.Category.warn(Category.java:1008)
        at
org.apache.hadoop.chukwa.datacollection.controller.ChukwaAgentController.addByName(ChukwaAgentController.java:267)
        at
org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender.subAppend(ChukwaDailyRollingFileAppender.java:565)
        - locked <0x00000000d6736ec0> (a java.lang.Object)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
        at
org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
        - locked <0x00000000d65b38c0> (a
com.glassdoor.chukwa.inputtools.log4j.GDChukwaDailyRollingFileAppender)
        at
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
        at org.apache.log4j.Category.callAppenders(Category.java:203)
        - locked <0x00000000d65c44b8> (a org.apache.log4j.Logger)
        at org.apache.log4j.Category.forcedLog(Category.java:388)
        at org.apache.log4j.Category.log(Category.java:853)
        at
org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:222)
        at
com.glassdoor.util.netsuite.service.NetSuiteServiceImpl.init(NetSuiteServiceImpl.java:86)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1378)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1339)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1299)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:463)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:404)
        at java.security.AccessController.doPrivileged(Native Method)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:375)
        at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:263)
        at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:170)
        - locked <0x00000000f77044f0> (a
java.util.concurrent.ConcurrentHashMap)
        at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:260) 
		at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:184)
        at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:163)
        at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:430)
        - locked <0x00000000f7704520> (a
java.util.concurrent.ConcurrentHashMap)
        at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
        at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
        - locked <0x00000000f77045d8> (a java.lang.Object)
        at
org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
        at
org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
        at
com.glassdoor.util.hadoop.entity.GDHadoopConfiguration.getAppContext(GDHadoopConfiguration.java:74)
        - locked <0x00000000d6736ef0> (a java.lang.Object)
        at
com.glassdoor.util.hadoop.entity.SpringHadoopMapperWrapper.run(SpringHadoopMapperWrapper.java:116)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
"communication thread":
        at
org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender.checkEntryConditions(ChukwaDailyRollingFileAppender.java:489)
        - waiting to lock <0x00000000d6736ec0> (a java.lang.Object)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:156)
        at
org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
        - locked <0x00000000d65b2c50> (a
com.glassdoor.chukwa.inputtools.log4j.GDChukwaDailyRollingFileAppender)
        at
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
        at org.apache.log4j.Category.callAppenders(Category.java:203)
        - locked <0x00000000d65c3e80> (a org.apache.log4j.Logger)
        at org.apache.log4j.Category.forcedLog(Category.java:388)
        at org.apache.log4j.Category.log(Category.java:853)
        at
org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:222)
        at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:554)
        at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.


I took a look at the code and surely, it is locking on chukwaLock object in
checkEntryConditions() and subAppend() methods. I'm not too sure if we need
to lock on chukwaLock for checkEntryConditions().
Anybody has a clue of what is going on ? This deadlock has happened twice
now so far.

Thanks,
Abhijit

--
View this message in context: http://apache-chukwa.679492.n3.nabble.com/Deadlock-happening-in-ChukwaDailyRollingFileAppender-tp4025029.html
Sent from the Chukwa - Users mailing list archive at Nabble.com.