cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prateek (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-6392) Cassandra 2.0.1 leaking open file handles for deleted system hints
Date Thu, 21 Nov 2013 22:53:35 GMT
Prateek created CASSANDRA-6392:
----------------------------------

             Summary: Cassandra 2.0.1 leaking open file handles for deleted system hints
                 Key: CASSANDRA-6392
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6392
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: Cassandra 2.0.1
Cassandra datastax java driver 1.0.3 (cassandra-driver-core)
            Reporter: Prateek


The system was running for a few days with a heavy write workload. The system crashed with
too many open file handles. It seems the deleted system hint file handles are still open even
though the file has been deleted on disk. We use a 3-node cluster with replication factor
3 on ec2 us-east-1 using NetworkTopologyStrategy. Default consistency level for write is ANY.

This is blocking our production deployment and causing instability in production. 

Appreciate your quick response.

Many thanks,
Prateek

See logs below:

[(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568 |wc -l
201499
[(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568 |grep deleted|wc
-l
186261

java    13568 ubuntu *676r   REG                9,0         356 1610775839 /mnt/cassandra/data/system/hints/system-hints-jb-848-Data.db
(deleted)
java    13568 ubuntu *677r   REG                9,0         309 1610770742 /mnt/cassandra/data/system/hints/system-hints-jb-845-Data.db
(deleted)
java    13568 ubuntu *678r   REG                9,0         506 1610775703 /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db
(deleted)


[(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# ls -l /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db
ls: cannot access /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db: No such file
or directory





--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message