incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Desimpel, Ignace" <Ignace.Desim...@nuance.com>
Subject Too many open files and stopped compaction with many pending compaction tasks
Date Thu, 27 Jun 2013 13:24:11 GMT
On a test with 3 cassandra servers version 1.2.5 with replication factor 1 and leveled compaction,
I did a store last night and I did not see any problem with Cassandra. On all 3 machine the
compaction is stopped already several hours. However , one machine reports 650 pending compaction
tasks (via jmx).
compaction_throughput_mb_per_sec is 0.
Concurrent_compactors is 3.
multithreaded_compaction = false.
No other load on these machines.

And when I start querying (using thrift), I get a 'too many open files' error on the machine
with pending compaction tasks.

Limits.conf setting for nofile is 65536
Using 'lsof'  and  'wc -l' I get a count of  59577 files for Cassandra.
Total count of keyspace files on disk : 20464.

The 3 machines have an equal (+/-) data load of about 60 GB. I see that 2 machines have no
unleveled or just 1 sstables on any keyspace, but on the machine with troubles there is one
keyspace having 670 unleveled sstables. Level sstable histo [670,28,106,14] thus 818 sstables.
An 'ls' on that directory counts for 5729 files, which corresponds to the 818 sstable (7 files
per sstables).

After restart of that machine I get 4037 open files for Cassandra. And also compaction has
restarted. Once finisched I get SSTableCountPerLEvel = [0,10, 109, 644].
Also, compaction reports speeds of 2.5 MB per sec. Seems slow too me. CPU less than 10%, Disk
15% with peeks to 45% (15000 rpm scsi). 14 GB free memory.

So I am puzzled about the number of open files and number of unleveled sstables, and a not
so fast compaction.

Anything than can be done? Or to be done so that the next time I can get more useful information?

Regards,
Ignace

Example output of lsof is :
java    10968 root  483r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  484u   REG                8,1  33554432 29229231 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568123.log
java    10968 root  485r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  486r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  487r   REG               8,17  39967253 14158943 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-481-Data.db
java    10968 root  488r   REG               8,17  58641524 14158942 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-481-Index.db
java    10968 root  489r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  490r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  491r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  492u   REG                8,1  33554432 29230501 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568134.log
java    10968 root  493r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  494r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  495r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  497u   REG                8,1  33554432 29242455 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568126.log
java    10968 root  498r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  499r   REG               8,17  39725539 14160146 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-1019-Data.db
java    10968 root  500r   REG               8,17  56369841 14160005 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-1019-Index.db
java    10968 root  502r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  504r   REG               8,17   1989198 14163384 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-922-Data.db
java    10968 root  505r   REG               8,17  40679209 14161763 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-543-Data.db
java    10968 root  506u   REG                8,1  33554432 29250917 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568106.log
java    10968 root  507r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  508r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  509r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  510r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
java    10968 root  511u   REG                8,1  33554432 29229238 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568108.log

Mime
View raw message