cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Smith <josh.sm...@careerbuilder.com>
Subject RE: C* files getting stuck
Date Thu, 30 Jun 2016 14:08:50 GMT
I have also faced this issue.  Rebooting the instance has been our fix so far.  I am very interested
if anyone else has a solution.  I was unable to get a definitive answer from Datastax during
the last Cassandra Summit.

From: Amit Singh F [mailto:amit.f.singh@ericsson.com]
Sent: Thursday, June 30, 2016 7:02 AM
To: user@cassandra.apache.org
Subject: RE: C* files getting stuck

Hi All,

Please check fi anybody has faced below issue and if yes what best can be done to avoid this.?
Thanks in advance.

Regards
Amit Singh

From: Amit Singh F [mailto:amit.f.singh@ericsson.com]
Sent: Wednesday, June 29, 2016 3:52 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: C* files getting stuck


Hi All

We are running Cassandra 2.0.14 and disk usage is very high. On investigating it further we
found that there are around 4-5 files(~ 150 GB) in stuck mode.

Command Fired : lsof /var/lib/cassandra | grep -i deleted

Output :

java 12158 cassandra 308r REG 8,16 34396638044 12727268 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-16481-Data.db
(deleted)
java 12158 cassandra 327r REG 8,16 101982374806 12715102 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-126861-Data.db
(deleted)
java 12158 cassandra 339r REG 8,16 12966304784 12714010 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-213548-Data.db
(deleted)
java 12158 cassandra 379r REG 8,16 15323318036 12714957 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-182936-Data.db
(deleted)

we are not able to see these files in any directory. This is somewhat similar to  https://issues.apache.org/jira/browse/CASSANDRA-6275
which is fixed but still issue is there on higher version. Also in logs no error related to
compaction is reported.

so could any one of you please provide any suggestion how to counter this. Restarting Cassandra
is one solution but this issue keeps on occurring so we cannot restart production machine
is not recommended so frequently.

Also we know that this version is not supported but there is high probability that it can
occur in higher version too.
Regards
Amit Singh

Mime
View raw message