cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ZhaoYang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-7145) FileNotFoundException during compaction
Date Mon, 15 Jun 2015 08:45:02 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585644#comment-14585644
] 

ZhaoYang edited comment on CASSANDRA-7145 at 6/15/15 8:44 AM:
--------------------------------------------------------------

encounter the same problem:  nodetool compact failed.

log: ERROR 08:38:42 Exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /home/zhaoyang/program/cassandra-2.1.1/./bin/../data/data/ycsb/usertable-4fb3447310e211e5a5c2f18cd916802f/ycsb-usertable-ka-6440-Data.db
(Too many open files)

at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1888)
~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67) ~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1674) ~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1686) ~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:275)
~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:324)
~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:151) ~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:465)
~[apache-cassandra-2.1.1.jar:2.1.1]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.1.jar:2.1.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_20]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_20]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_20]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_20]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]


I was doing YCSB testing. 

Setup: C* 2.1.1, Ubuntu 14.04 LTS, 24GB Ram, 256 ssd. 


was (Author: jasonstack):
encounter the same problem:  nodetool compact failed.

log: ERROR 08:38:42 Exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /home/zhaoyang/program/cassandra-2.1.1/./bin/../data/data/ycsb/usertable-4fb3447310e211e5a5c2f18cd916802f/ycsb-usertable-ka-6440-Data.db
(Too many open files)

I was doing YCSB testing. 

Setup: C* 2.1.1, Ubuntu 14.04 LTS, 24GB Ram, 256 ssd. 

> FileNotFoundException during compaction
> ---------------------------------------
>
>                 Key: CASSANDRA-7145
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7145
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: CentOS 6.3, Datastax Enterprise 4.0.1 (Cassandra 2.0.5), Java 1.7.0_55
>            Reporter: PJ
>            Assignee: Marcus Eriksson
>             Fix For: 1.2.19, 2.0.11, 2.1.0
>
>         Attachments: 0001-avoid-marking-compacted-sstables-as-compacting.patch, compaction
- FileNotFoundException.txt, repair - RuntimeException.txt, startup - AssertionError.txt
>
>
> I can't finish any compaction because my nodes always throw a "FileNotFoundException".
I've already tried the following but nothing helped:
> 1. nodetool flush
> 2. nodetool repair (ends with RuntimeException; see attachment)
> 3. node restart (via dse cassandra-stop)
> Whenever I restart the nodes, another type of exception is logged (see attachment) somewhere
near the end of startup process. This particular exception doesn't seem to be critical because
the nodes still manage to finish the startup and become online.
> I don't have specific steps to reproduce the problem that I'm experiencing with compaction
and repair. I'm in the middle of migrating 4.8 billion rows from MySQL via SSTableLoader.

> Some things that may or may not be relevant:
> 1. I didn't drop and recreate the keyspace (so probably not related to CASSANDRA-4857)
> 2. I do the bulk-loading in batches of 1 to 20 millions rows. When a batch reaches 100%
total progress (i.e. starts to build secondary index), I kill the sstableloader process and
cancel the index build
> 3. I restart the nodes occasionally. It's possible that there is an on-going compaction
during one of those restarts.
> Related StackOverflow question (mine): http://stackoverflow.com/questions/23435847/filenotfoundexception-during-compaction



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message