cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CASSANDRA-6392) Cassandra 2.0.1 leaking open file handles for deleted system hints
Date Thu, 21 Nov 2013 23:39:35 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis resolved CASSANDRA-6392.
---------------------------------------

    Resolution: Duplicate

> Cassandra 2.0.1 leaking open file handles for deleted system hints
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-6392
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6392
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 2.0.1
> Cassandra datastax java driver 1.0.3 (cassandra-driver-core)
>            Reporter: Prateek
>
> The system was running for a few days with a heavy write workload. The system crashed
with too many open file handles. It seems the deleted system hint file handles are still open
even though the file has been deleted on disk. We use a 3-node cluster with replication factor
3 on ec2 us-east-1 using NetworkTopologyStrategy. Default consistency level for write is ANY.
> This is blocking our production deployment and causing instability in production. 
> Appreciate your quick response.
> Many thanks,
> Prateek
> See logs below:
> [(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568 |wc -l
> 201499
> [(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568 |grep deleted|wc
-l
> 186261
> java    13568 ubuntu *676r   REG                9,0         356 1610775839 /mnt/cassandra/data/system/hints/system-hints-jb-848-Data.db
(deleted)
> java    13568 ubuntu *677r   REG                9,0         309 1610770742 /mnt/cassandra/data/system/hints/system-hints-jb-845-Data.db
(deleted)
> java    13568 ubuntu *678r   REG                9,0         506 1610775703 /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db
(deleted)
> [(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# ls -l /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db
> ls: cannot access /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db: No such
file or directory



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message