accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (ACCUMULO-1110) deadlock found in major compaction
Date Thu, 28 Feb 2013 21:03:12 GMT

     [ https://issues.apache.org/jira/browse/ACCUMULO-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eric Newton resolved ACCUMULO-1110.
-----------------------------------

    Resolution: Fixed
    
> deadlock found in major compaction
> ----------------------------------
>
>                 Key: ACCUMULO-1110
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1110
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>         Environment: ten-node test cluster running random walk test
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>            Priority: Critical
>              Labels: 15_qa_bug
>             Fix For: 1.5.0
>
>
> {noformat}
> Found one Java-level deadlock:
> =============================
> "ClientPool 131":
>   waiting to lock monitor 0x000000005e264210 (object 0x0000000765399a38, a org.apache.accumulo.server.tabletserver.Tablet),
>   which is held by "ClientPool 76"
> "ClientPool 76":
>   waiting to lock monitor 0x0000000061ed94d0 (object 0x000000075a3a0478, a org.apache.accumulo.server.tabletserver.CompactionQueue),
>   which is held by "major compactor 3"
> "major compactor 3":
>   waiting to lock monitor 0x00002aaac470b690 (object 0x0000000764d0d688, a org.apache.accumulo.server.tabletserver.Tablet),
>   which is held by "ClientPool 36"
> "ClientPool 36":
>   waiting to lock monitor 0x0000000061ed94d0 (object 0x000000075a3a0478, a org.apache.accumulo.server.tabletserver.CompactionQueue),
>   which is held by "major compactor 3"
> Java stack information for the threads listed above:
> ===================================================
> "ClientPool 131":
> 	at org.apache.accumulo.server.tabletserver.Tablet.importMapFiles(Tablet.java:3666)
> 	- waiting to lock <0x0000000765399a38> (a org.apache.accumulo.server.tabletserver.Tablet)
> 	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.bulkImport(TabletServer.java:898)
> 	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
> 	at $Proxy2.bulkImport(Unknown Source)
> 	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2153)
> 	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2137)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:156)
> 	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> 	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:208)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> 	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> 	at java.lang.Thread.run(Thread.java:662)
> "ClientPool 76":
> 	at org.apache.accumulo.server.tabletserver.CompactionQueue.offer(CompactionQueue.java:54)
> 	- waiting to lock <0x000000075a3a0478> (a org.apache.accumulo.server.tabletserver.CompactionQueue)
> 	at org.apache.accumulo.server.tabletserver.CompactionQueue.offer(CompactionQueue.java:28)
> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:653)
> 	at org.apache.accumulo.trace.instrument.TraceExecutorService.execute(TraceExecutorService.java:39)
> 	at org.apache.accumulo.server.tabletserver.TabletServerResourceManager.executeMajorCompaction(TabletServerResourceManager.java:717)
> 	at org.apache.accumulo.server.tabletserver.TabletServerResourceManager$TabletResourceManager.executeMajorCompaction(TabletServerResourceManager.java:694)
> 	at org.apache.accumulo.server.tabletserver.Tablet.initiateMajorCompaction(Tablet.java:2924)
> 	- locked <0x0000000765399a38> (a org.apache.accumulo.server.tabletserver.Tablet)
> 	at org.apache.accumulo.server.tabletserver.Tablet.importMapFiles(Tablet.java:368)
> 	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.bulkImport(TabletServer.java:898)
> 	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
> 	at $Proxy2.bulkImport(Unknown Source)
> 	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2153)
> 	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2137)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:156)
> 	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> 	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:208)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> 	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> 	at java.lang.Thread.run(Thread.java:662)
> "major compactor 3":
> 	at org.apache.accumulo.server.tabletserver.Tablet$CompactionRunner.getNumFiles(Tablet.java:2895)
> 	- waiting to lock <0x0000000764d0d688> (a org.apache.accumulo.server.tabletserver.Tablet)
> 	at org.apache.accumulo.server.tabletserver.Tablet$CompactionRunner.compareTo(Tablet.java:2912)
> 	at org.apache.accumulo.server.tabletserver.Tablet$CompactionRunner.compareTo(Tablet.java:2848)
> 	at org.apache.accumulo.trace.instrument.TraceRunnable.compareTo(TraceRunnable.java:54)
> 	at org.apache.accumulo.trace.instrument.TraceRunnable.compareTo(TraceRunnable.java:23)
> 	at java.util.Collections.min(Collections.java:574)
> 	at org.apache.accumulo.server.tabletserver.CompactionQueue.poll(CompactionQueue.java:38)
> 	- locked <0x000000075a3a0478> (a org.apache.accumulo.server.tabletserver.CompactionQueue)
> 	at org.apache.accumulo.server.tabletserver.CompactionQueue.take(CompactionQueue.java:78)
> 	- locked <0x000000075a3a0478> (a org.apache.accumulo.server.tabletserver.CompactionQueue)
> 	at org.apache.accumulo.server.tabletserver.CompactionQueue.take(CompactionQueue.java:28)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> 	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> 	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> 	at java.lang.Thread.run(Thread.java:662)
> "ClientPool 36":
> 	at org.apache.accumulo.server.tabletserver.CompactionQueue.offer(CompactionQueue.java:54)
> 	- waiting to lock <0x000000075a3a0478> (a org.apache.accumulo.server.tabletserver.CompactionQueue)
> 	at org.apache.accumulo.server.tabletserver.CompactionQueue.offer(CompactionQueue.java:28)
> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:653)
> 	at org.apache.accumulo.trace.instrument.TraceExecutorService.execute(TraceExecutorService.java:39)
> 	at org.apache.accumulo.server.tabletserver.TabletServerResourceManager.executeMajorCompaction(TabletServerResourceManager.java:717)
> 	at org.apache.accumulo.server.tabletserver.TabletServerResourceManager$TabletResourceManager.executeMajorCompaction(TabletServerResourceManager.java:694)
> 	at org.apache.accumulo.server.tabletserver.Tablet.initiateMajorCompaction(Tablet.java:2924)
> 	- locked <0x0000000764d0d688> (a org.apache.accumulo.server.tabletserver.Tablet)
> 	at org.apache.accumulo.server.tabletserver.Tablet.compactAll(Tablet.java:3871)
> 	- locked <0x0000000764d0d688> (a org.apache.accumulo.server.tabletserver.Tablet)
> 	at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.compact(TabletServer.java:2048)
> 	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
> 	at $Proxy2.compact(Unknown Source)
> 	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$compact.getResult(TabletClientService.java:2296)
> 	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$compact.getResult(TabletClientService.java:2282)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:156)
> 	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> 	at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:208)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> 	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> 	at java.lang.Thread.run(Thread.java:662)
> Found 1 deadlock.
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message