Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3F9DBD991 for ; Fri, 8 Mar 2013 05:06:17 +0000 (UTC) Received: (qmail 46335 invoked by uid 500); 8 Mar 2013 05:06:16 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 45445 invoked by uid 500); 8 Mar 2013 05:06:15 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 45229 invoked by uid 99); 8 Mar 2013 05:06:15 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Mar 2013 05:06:15 +0000 Date: Fri, 8 Mar 2013 05:06:15 +0000 (UTC) From: "Jonathan Ellis (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-5318) Exception in CompactionExecutor thread after nodetool drain MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-5318?page=3Dcom.atlas= sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D= 13596831#comment-13596831 ]=20 Jonathan Ellis commented on CASSANDRA-5318: ------------------------------------------- You do know that's what drain is supposed to do, right? =20 > Exception in CompactionExecutor thread after nodetool drain > ----------------------------------------------------------- > > Key: CASSANDRA-5318 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5318 > Project: Cassandra > Issue Type: Bug > Affects Versions: 1.2.2 > Environment: I am running cluster of 6 nodes in amazon(2 nodes pe= r availability zone in eu-west region) using EC2snith, Cassandra 1.2.2 on u= buntu 12.04, m1.xlarge and 4 x 450 GB of ephemeral volumes stripped togethe= r on each node. > Reporter: Michael Czerwi=C5=84ski > > I am doing batch inserts around 5K rows per batch, 400 writes per second. > I am using leveled compaction and multithreaded_compaction enabled. > After I do "nodetool drain" I see in the logs memtables being flushed aft= er which I see exception in CompactionExecutor Thread. > INFO 15:20:17,471 Completed flushing /mnt/cassandra/data/OpsCenter/pdps/O= psCenter-pdps-ib-60-Data.db (316475 bytes) for commitlog position ReplayPos= ition(segmentId=3D1362668614841, position=3D26678466) > INFO 15:20:17,472 Writing Memtable-rollups300@466350667(305256/9115571 s= erialized/live bytes, 12719 ops) > INFO 15:20:17,568 Completed flushing /mnt/cassandra/data/OpsCenter/rollu= ps300/OpsCenter-rollups300-ib-60-Data.db (275536 bytes) for commitlog posit= ion ReplayPosition(segmentId=3D1362668614841, position=3D26678466) > INFO 15:20:17,687 DRAINED > ERROR 15:20:18,271 Exception in thread Thread[CompactionExecutor:15,1,RMI= Runtime] > java.util.concurrent.RejectedExecutionException > at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedEx= ecution(ThreadPoolExecutor.java:1768) > at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecu= tor.java:767) > at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecut= e(ScheduledThreadPoolExecutor.java:215) > at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Sche= duledThreadPoolExecutor.java:397) > at java.util.concurrent.ScheduledThreadPoolExecutor.submit(Schedu= ledThreadPoolExecutor.java:470) > at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(S= STableDeletingTask.java:63) > at org.apache.cassandra.io.sstable.SSTableReader.releaseReference= (SSTableReader.java:902) > at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(Data= Tracker.java:361) > at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.ja= va:330) > at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:3= 24) > at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(D= ataTracker.java:227) > at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTa= bles(ColumnFamilyStore.java:997) > at org.apache.cassandra.db.compaction.CompactionTask.runWith(Comp= actionTask.java:231) > at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(Dis= kAwareRunnable.java:48) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:28) > at org.apache.cassandra.db.compaction.CompactionTask.executeInter= nal(CompactionTask.java:59) > at org.apache.cassandra.db.compaction.AbstractCompactionTask.exec= ute(AbstractCompactionTask.java:62) > at org.apache.cassandra.db.compaction.CompactionManager$Backgroun= dCompactionTask.run(CompactionManager.java:191) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadP= oolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR 15:20:21,102 Exception in thread Thread[CompactionExecutor:14,1,RMI= Runtime] > java.util.concurrent.RejectedExecutionException > at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedEx= ecution(ThreadPoolExecutor.java:1768) > at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecu= tor.java:767) > at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecut= e(ScheduledThreadPoolExecutor.java:215) > at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Sche= duledThreadPoolExecutor.java:397) > at java.util.concurrent.ScheduledThreadPoolExecutor.submit(Schedu= ledThreadPoolExecutor.java:470) > at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(S= STableDeletingTask.java:63) > at org.apache.cassandra.io.sstable.SSTableReader.releaseReference= (SSTableReader.java:902) > at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(Data= Tracker.java:361) > at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.ja= va:330) > at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:3= 24) > at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(D= ataTracker.java:227) > at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTa= bles(ColumnFamilyStore.java:997) > at org.apache.cassandra.db.compaction.CompactionTask.runWith(Comp= actionTask.java:231) > at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(Dis= kAwareRunnable.java:48) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:28) > at org.apache.cassandra.db.compaction.CompactionTask.executeInter= nal(CompactionTask.java:59) > at org.apache.cassandra.db.compaction.AbstractCompactionTask.exec= ute(AbstractCompactionTask.java:62) > at org.apache.cassandra.db.compaction.CompactionManager$Backgroun= dCompactionTask.run(CompactionManager.java:191) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadP= oolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR 15:20:22,760 Exception in thread Thread[CompactionExecutor:13,1,RMI= Runtime] > java.util.concurrent.RejectedExecutionException > at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedEx= ecution(ThreadPoolExecutor.java:1768) > at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecu= tor.java:767) > at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecut= e(ScheduledThreadPoolExecutor.java:215) > at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Sche= duledThreadPoolExecutor.java:397) > at java.util.concurrent.ScheduledThreadPoolExecutor.submit(Schedu= ledThreadPoolExecutor.java:470) > at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(S= STableDeletingTask.java:63) > at org.apache.cassandra.io.sstable.SSTableReader.releaseReference= (SSTableReader.java:902) > at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(Data= Tracker.java:361) > at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.ja= va:330) > at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:3= 24) > at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(D= ataTracker.java:227) > at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTa= bles(ColumnFamilyStore.java:997) > at org.apache.cassandra.db.compaction.CompactionTask.runWith(Comp= actionTask.java:231) > at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(Dis= kAwareRunnable.java:48) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:28) > at org.apache.cassandra.db.compaction.CompactionTask.executeInter= nal(CompactionTask.java:59) > at org.apache.cassandra.db.compaction.AbstractCompactionTask.exec= ute(AbstractCompactionTask.java:62) > at org.apache.cassandra.db.compaction.CompactionManager$Backgroun= dCompactionTask.run(CompactionManager.java:191) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadP= oolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR 15:20:24,587 Exception in thread Thread[CompactionExecutor:16,1,RMI= Runtime] > java.util.concurrent.RejectedExecutionException > at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedEx= ecution(ThreadPoolExecutor.java:1768) > at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecu= tor.java:767) > at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecut= e(ScheduledThreadPoolExecutor.java:215) > at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Sche= duledThreadPoolExecutor.java:397) > at java.util.concurrent.ScheduledThreadPoolExecutor.submit(Schedu= ledThreadPoolExecutor.java:470) > at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(S= STableDeletingTask.java:63) > at org.apache.cassandra.io.sstable.SSTableReader.releaseReference= (SSTableReader.java:902) > at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(Data= Tracker.java:361) > at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.ja= va:330) > at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:3= 24) > at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(D= ataTracker.java:227) > at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTa= bles(ColumnFamilyStore.java:997) > at org.apache.cassandra.db.compaction.CompactionTask.runWith(Comp= actionTask.java:231) > at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(Dis= kAwareRunnable.java:48) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable= .java:28) > at org.apache.cassandra.db.compaction.CompactionTask.executeInter= nal(CompactionTask.java:59) > at org.apache.cassandra.db.compaction.AbstractCompactionTask.exec= ute(AbstractCompactionTask.java:62) > at org.apache.cassandra.db.compaction.CompactionManager$Backgroun= dCompactionTask.run(CompactionManager.java:191) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadP= oolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > This is replicable for me under this load by simply running nodetool drai= n. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrato= rs For more information on JIRA, see: http://www.atlassian.com/software/jira