incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Hanna <jeremy.hanna1...@gmail.com>
Subject Re: Too many open files and stopped compaction with many pending compaction tasks
Date Thu, 27 Jun 2013 13:35:57 GMT
Are you on SSDs?

On 27 Jun 2013, at 14:24, "Desimpel, Ignace" <Ignace.Desimpel@nuance.com> wrote:

> On a test with 3 cassandra servers version 1.2.5 with replication factor 1 and leveled
compaction, I did a store last night and I did not see any problem with Cassandra. On all
3 machine the compaction is stopped already several hours. However , one machine reports 650
pending compaction tasks (via jmx).
> compaction_throughput_mb_per_sec is 0.
> Concurrent_compactors is 3.
> multithreaded_compaction = false.
> No other load on these machines.
>  
> And when I start querying (using thrift), I get a ’too many open files’ error on
the machine with pending compaction tasks.
>  
> Limits.conf setting for nofile is 65536
> Using ‘lsof’  and  ‘wc –l’ I get a count of  59577 files for Cassandra.
> Total count of keyspace files on disk : 20464.
>  
> The 3 machines have an equal (+/-) data load of about 60 GB. I see that 2 machines have
no unleveled or just 1 sstables on any keyspace, but on the machine with troubles there is
one keyspace having 670 unleveled sstables. Level sstable histo [670,28,106,14] thus 818 sstables.
An ‘ls’ on that directory counts for 5729 files, which corresponds to the 818 sstable
(7 files per sstables).
>  
> After restart of that machine I get 4037 open files for Cassandra. And also compaction
has restarted. Once finisched I get SSTableCountPerLEvel = [0,10, 109, 644].
> Also, compaction reports speeds of 2.5 MB per sec. Seems slow too me. CPU less than 10%,
Disk 15% with peeks to 45% (15000 rpm scsi). 14 GB free memory.
>  
> So I am puzzled about the number of open files and number of unleveled sstables, and
a not so fast compaction.
>  
> Anything than can be done? Or to be done so that the next time I can get more useful
information?
>  
> Regards,
> Ignace
>  
> Example output of lsof is :
> java    10968 root  483r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  484u   REG                8,1  33554432 29229231 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568123.log
> java    10968 root  485r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  486r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  487r   REG               8,17  39967253 14158943 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-481-Data.db
> java    10968 root  488r   REG               8,17  58641524 14158942 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-481-Index.db
> java    10968 root  489r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  490r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  491r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  492u   REG                8,1  33554432 29230501 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568134.log
> java    10968 root  493r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  494r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  495r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  497u   REG                8,1  33554432 29242455 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568126.log
> java    10968 root  498r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  499r   REG               8,17  39725539 14160146 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-1019-Data.db
> java    10968 root  500r   REG               8,17  56369841 14160005 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-1019-Index.db
> java    10968 root  502r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  504r   REG               8,17   1989198 14163384 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-922-Data.db
> java    10968 root  505r   REG               8,17  40679209 14161763 /media/datadrive1/dbdatafile/Ks100K/ReverseStringFunction/Ks100K-ReverseStringFunction-ic-543-Data.db
> java    10968 root  506u   REG                8,1  33554432 29250917 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568106.log
> java    10968 root  507r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  508r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  509r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  510r   REG               8,17  10507031 14156174 /media/datadrive1/dbdatafile/Ks100K/ReverseLabelFunction/Ks100K-ReverseLabelFunction-ic-1512-Data.db
> java    10968 root  511u   REG                8,1  33554432 29229238 /home/cassandra/deployed/data/cdi.cassandra.cdi/dbcommitlog/CommitLog-2-1372260568108.log


Mime
View raw message