Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 25C114EE6 for ; Fri, 17 Jun 2011 17:07:44 +0000 (UTC) Received: (qmail 13798 invoked by uid 500); 17 Jun 2011 17:07:41 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 13769 invoked by uid 500); 17 Jun 2011 17:07:41 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 13761 invoked by uid 99); 17 Jun 2011 17:07:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:07:41 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ryan@twitter.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-iw0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:07:37 +0000 Received: by iwn39 with SMTP id 39so2902216iwn.31 for ; Fri, 17 Jun 2011 10:07:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=twitter.com; s=google; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type:content-transfer-encoding; bh=PO7o9+IipLxerj7ybuWVDo/sL1G7K21LmlZWg7s5IU4=; b=IN229bteyhABqJ36FeJwZbLMfPO6qcWeW4Nf4z9gR+Pd7S6bhii5tWv53t2ZN2D8Dn VXmtzWDk0Cj8svUItlVXdiChd6z2AYxJW54/wcvjvEHx8/1nMoWGNy0IaswE6U6pq7Ne GxeA/uEeabzCgjACkvtu5FUf8bxKwU0taVidM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=twitter.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=Jh/aZvXSdzqlbzoYa0p4e8eVmnaWKEGvIw9NgC3XipqClT1otYwAhz/QNW1kWL2L9n skLinVdffsQ7cQWC59NMk12LRRZD/I7WOn30mlwThAtlkXIoUs0gM1znH5l4Ge+hePU8 YMG4t3lmH4Rfns0N87lGs2O4gSzcEUG4oEXb8= Received: by 10.43.135.72 with SMTP id if8mr1267491icc.265.1308330436108; Fri, 17 Jun 2011 10:07:16 -0700 (PDT) MIME-Version: 1.0 Received: by 10.42.219.8 with HTTP; Fri, 17 Jun 2011 10:06:56 -0700 (PDT) In-Reply-To: References: From: Ryan King Date: Fri, 17 Jun 2011 10:06:56 -0700 Message-ID: Subject: Re: SSTable corruption blocking compaction and scrub can't fix it To: user@cassandra.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Even without lsof, you should be able to get the data from /proc/$pid -ryan On Fri, Jun 17, 2011 at 5:08 AM, Dominic Williams wrote: > Unfortunately I shutdown that node and anyway lsof wasn't installed. > But $ulimit gives > unlimited > > On 17 June 2011 13:00, Sylvain Lebresne wrote: >> >> On Fri, Jun 17, 2011 at 1:51 PM, Dominic Williams >> wrote: >> > As far as scrub goes that could be it. I'm already running unlimited >> > file >> > handles though so ulimit not answer unfortunately >> >> Are you sure ? How many file descriptors are open on the system when >> you get that >> scrub exception ? If you really have unlimited file handles and >> Cassandra uses them >> all, then we have a more serious problem than just CASSANDRA-2669. >> >> -- >> Sylvain >> >> > Dominic >> > On 17 June 2011 12:12, Sylvain Lebresne wrote: >> >> >> >> Scrub apparently dies because it cannot acquire a file descriptor. >> >> Scrub >> >> does >> >> not correctly closes files >> >> (https://issues.apache.org/jira/browse/CASSANDRA-2669) >> >> so that may be part of why that happens. However, a simple fix is >> >> probably >> >> to >> >> raise up the file descriptor limit. >> >> >> >> -- >> >> Sylvain >> >> >> >> >> >> On Fri, Jun 17, 2011 at 11:31 AM, Dominic Williams >> >> wrote: >> >> > Hi all, >> >> > Anyone experiencing this..? >> >> > I noticed one of my 7.6-2 nodes had inexplicable and consistently >> >> > high >> >> > cpu >> >> > usage. Checking the log I found that there was a some kind of SSTab= le >> >> > corruption that was stopping a bunch of files from compacting (firs= t >> >> > trace >> >> > copied below). >> >> > I then tried scrub (before anyone asks it had been run when >> >> > upgrading, >> >> > but I >> >> > was just trying to find out if this fixed it) but that failed with >> >> > "Too >> >> > many >> >> > open files" even though node should be using mem map (second trace >> >> > copied >> >> > below). >> >> > Current plan is to snapshot, decommission & bootstrap but not what >> >> > should be >> >> > happening. Only factor I can think of that might have caused proble= m >> >> > is >> >> > we >> >> > are currently running huge GCGraceSeconds because we can't run repa= ir >> >> > (we >> >> > are hanging on for 8.1 as CASSANDRA-2280 causes repair to consume a= ll >> >> > disk >> >> > space and fail) >> >> > =C2=A0INFO [NonPeriodicTasks:1] 2011-06-16 22:43:45,232 SSTable.jav= a (line >> >> > 147) >> >> > Deleted /var/opt/cassandra/data/FightMyMonster/Users_CisIndex-f-592= 2 >> >> > =C2=A0INFO [MutationStage:7989] 2011-06-16 22:49:57,852 >> >> > ColumnFamilyStore.java >> >> > (line 1065) Enqueuing flush of Memtable-UserMonsters@423965881(3447= 1 >> >> > bytes, >> >> > 322112 operations) >> >> > =C2=A0INFO [FlushWriter:1176] 2011-06-16 22:49:57,917 Memtable.java= (line >> >> > 157) >> >> > Writing Memtable-UserMonsters@423965881(34471 bytes, 322112 >> >> > operations) >> >> > =C2=A0INFO [FlushWriter:1176] 2011-06-16 22:50:01,087 Memtable.java= (line >> >> > 172) >> >> > Completed flushing >> >> > /var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7614-Data.db >> >> > (7250239 >> >> > bytes) >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-16 22:50:01,090 >> >> > CompactionManager.java >> >> > (line 395) Compacting >> >> > >> >> > >> >> > [SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserM= onsters-f-6993-Data.db'),SSTableReader( >> >> > >> >> > >> >> > path=3D'/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-6994-= Data.db'),SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/User= Monsters-f-6995-Data.db'),SSTableReader(path=3D'/var/opt/cassandra >> >> > >> >> > >> >> > /data/FightMyMonster/UserMonsters-f-6996-Data.db'),SSTableReader(pa= th=3D'/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-6998-Data.db'),= SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/Use >> >> > >> >> > >> >> > rMonsters-f-7000-Data.db'),SSTableReader(path=3D'/var/opt/cassandra= /data/FightMyMonster/UserMonsters-f-7002-Data.db'),SSTableReader(path=3D'/v= ar/opt/cassandra/data/FightMyMonster/UserMonsters-f-7004-Data.db >> >> > >> >> > >> >> > '),SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/Use= rMonsters-f-7006-Data.db'),SSTableReader(path=3D'/var/opt/cassandra/data/Fi= ghtMyMonster/UserMonsters-f-7008-Data.db'),SSTableReader(path=3D'/ >> >> > >> >> > >> >> > var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7010-Data.db')= ,SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMonsters-= f-7012-Data.db'),SSTableReader(path=3D'/var/opt/cassandra/data/F >> >> > >> >> > >> >> > ightMyMonster/UserMonsters-f-7014-Data.db'),SSTableReader(path=3D'/= var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7016-Data.db'),SSTable= Reader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMonste >> >> > >> >> > >> >> > rs-f-7018-Data.db'),SSTableReader(path=3D'/var/opt/cassandra/data/F= ightMyMonster/UserMonsters-f-7020-Data.db'),SSTableReader(path=3D'/var/opt/= cassandra/data/FightMyMonster/UserMonsters-f-7022-Data.db'),SSTa >> >> > >> >> > >> >> > bleReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMonste= rs-f-7024-Data.db'),SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMo= nster/UserMonsters-f-7026-Data.db'),SSTableReader(path=3D'/var/opt >> >> > >> >> > >> >> > /cassandra/data/FightMyMonster/UserMonsters-f-7028-Data.db'),SSTabl= eReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7030-= Data.db'),SSTableReader(path=3D'/var/opt/cassandra/data/FightMyM >> >> > >> >> > >> >> > onster/UserMonsters-f-7032-Data.db'),SSTableReader(path=3D'/var/opt= /cassandra/data/FightMyMonster/UserMonsters-f-7034-Data.db'),SSTableReader(= path=3D'/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-70 >> >> > >> >> > >> >> > 36-Data.db'),SSTableReader(path=3D'/var/opt/cassandra/data/FightMyM= onster/UserMonsters-f-7038-Data.db'),SSTableReader(path=3D'/var/opt/cassand= ra/data/FightMyMonster/UserMonsters-f-7040-Data.db'),SSTableRead >> >> > >> >> > >> >> > er(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-70= 42-Data.db'),SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/U= serMonsters-f-7044-Data.db'),SSTableReader(path=3D'/var/opt/cassan >> >> > >> >> > >> >> > dra/data/FightMyMonster/UserMonsters-f-7046-Data.db'),SSTableReader= (path=3D'/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7048-Data.db= '),SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/ >> >> > >> >> > >> >> > UserMonsters-f-7050-Data.db'),SSTableReader(path=3D'/var/opt/cassan= dra/data/FightMyMonster/UserMonsters-f-7052-Data.db')] >> >> > =C2=A0INFO [NonPeriodicTasks:1] 2011-06-16 22:50:02,399 >> >> > ColumnFamilyStore.java >> >> > (line 1065) Enqueuing flush of Memtable-Monsters@562380513(497561 >> >> > bytes, >> >> > 13860 operations) >> >> > =C2=A0INFO [FlushWriter:1176] 2011-06-16 22:50:02,399 Memtable.java= (line >> >> > 157) >> >> > Writing Memtable-Monsters@562380513(497561 bytes, 13860 operations) >> >> > =C2=A0INFO [FlushWriter:1176] 2011-06-16 22:50:02,580 Memtable.java= (line >> >> > 172) >> >> > Completed flushing >> >> > /var/opt/cassandra/data/FightMyMonster/Monsters-f-7160-Data.db >> >> > (471100 >> >> > bytes) >> >> > ERROR [CompactionExecutor:1] 2011-06-16 22:50:37,776 >> >> > AbstractCassandraDaemon.java (line 114) Fatal exception in thread >> >> > Thread[CompactionExecutor:1,1,main] >> >> > java.io.IOError: java.io.IOException: Corrupt (negative) value leng= th >> >> > encountered >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnS= ortedMap.java:252) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.ja= va:268) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.ja= va:227) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(Concurre= ntSkipListMap.java:1493) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > java.util.concurrent.ConcurrentSkipListMap.(ConcurrentSkipLis= tMap.java:1443) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColu= mn.java:379) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColu= mn.java:362) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColu= mn.java:322) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(C= olumnFamilySerializer.java:129) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFa= milyWithColumns(SSTableIdentityIterator.java:201) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > org.apache.cassandra.io.PrecompactedRow.(PrecompactedRow.java= :78) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.io.CompactionIterator.getCompactedRow(Compacti= onIterator.java:154) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIte= rator.java:110) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> >> > >> >> > >> >> > org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIte= rator.java:45) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> >> > >> >> > >> >> > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIte= rator.java:74) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > com.google.common.collect.AbstractIterator.tryToComputeNext(Abstrac= tIterator.java:136) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0at >> >> > >> >> > >> >> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator= .java:131) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.commons.collections.iterators.FilterIterator.setNextObje= ct(FilterIterator.java:183) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.commons.collections.iterators.FilterIterator.hasNext(Fil= terIterator.java:94) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.CompactionManager.doCompaction(CompactionMa= nager.java:448) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.= java:124) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.= java:94) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.FutureTask.run(= FutureTask.java:138) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolEx= ecutor.java:886) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:908) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.run(Thread.java:619= ) >> >> > Caused by: java.io.IOException: Corrupt (negative) value length >> >> > encountered >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBuffer= Util.java:315) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializ= er.java:99) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnS= ortedMap.java:248) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 ... 26 more >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-16 22:50:37,794 >> >> > CompactionManager.java >> >> > (line 395) Compacting >> >> > >> >> > >> >> > [SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/Monst= ers-f-7157-Data.db'),SSTableReader(path >> >> > >> >> > >> >> > =3D'/var/opt/cassandra/data/FightMyMonster/Monsters-f-7158-Data.db'= ),SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/Monsters-f-7= 159-Data.db'),SSTableReader(path=3D'/var/opt/cassandra/data/FightM >> >> > yMonster/Monsters-f-7160-Data.db')] >> >> > >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:24,756 >> >> > CompactionManager.java >> >> > (line 652) Scrub of >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7554-Data.db') >> >> > complete: 244 rows in new sstable and 0 empty (tombstoned) rows >> >> > dropped >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:24,756 >> >> > CompactionManager.java >> >> > (line 511) Scrubbing >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7446-Data.db') >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:26,115 >> >> > CompactionManager.java >> >> > (line 652) Scrub of >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7446-Data.db') >> >> > complete: 493 rows in new sstable and 0 empty (tombstoned) rows >> >> > dropped >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:26,115 >> >> > CompactionManager.java >> >> > (line 511) Scrubbing >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7018-Data.db') >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:26,225 >> >> > CompactionManager.java >> >> > (line 652) Scrub of >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7018-Data.db') >> >> > complete: 275 rows in new sstable and 0 empty (tombstoned) rows >> >> > dropped >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:26,226 >> >> > CompactionManager.java >> >> > (line 511) Scrubbing >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7580-Data.db') >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:28,383 >> >> > CompactionManager.java >> >> > (line 652) Scrub of >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7580-Data.db') >> >> > complete: 297 rows in new sstable and 0 empty (tombstoned) rows >> >> > dropped >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:28,384 >> >> > CompactionManager.java >> >> > (line 511) Scrubbing >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7574-Data.db') >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 >> >> > CompactionManager.java >> >> > (line 652) Scrub of >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7574-Data.db') >> >> > complete: 347 rows in new sstable and 0 empty (tombstoned) rows >> >> > dropped >> >> > =C2=A0INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 >> >> > CompactionManager.java >> >> > (line 511) Scrubbing >> >> > >> >> > >> >> > SSTableReader(path=3D'/var/opt/cassandra/data/FightMyMonster/UserMo= nsters-f-7010-Data.db') >> >> > ERROR [CompactionExecutor:1] 2011-06-17 00:46:29,374 >> >> > AbstractCassandraDaemon.java (line 114) Fatal exception in thread >> >> > Thread[CompactionExecutor:1,1,main] >> >> > java.io.FileNotFoundException: >> >> > >> >> > /var/opt/cassandra/data/FightMyMonster/UserMonsters-tmp-f-7823-Data= .db >> >> > (Too >> >> > many open files) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.io.RandomAccessFile.open(Native= Method) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.io.RandomAccessFile.(Rand= omAccessFile.java:212) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.io.util.BufferedRandomAccessFile.(Buffer= edRandomAccessFile.java:113) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.io.sstable.SSTableWriter.(SSTableWriter.= java:78) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(Co= lumnFamilyStore.java:2243) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.CompactionManager.maybeCreateWriter(Compact= ionManager.java:794) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager= .java:534) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.CompactionManager.access$600(CompactionMana= ger.java:56) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.= java:195) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.FutureTask.run(= FutureTask.java:138) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolEx= ecutor.java:886) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at >> >> > >> >> > >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:908) >> >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.run(Thread.java:619= ) >> >> > Dominic >> > >> > > >