incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Strange Behavior : Commitlog data is not flushed
Date Wed, 06 Oct 2010 19:01:16 GMT
Commitlog segments remain until all the data in them has been flushed.
 Reduce MemtableFlushAfterMinutes.

If I had to guess without your error log why the node went down, I
would guess you exceeded the open file handle allowance.  You can
increase that with the standard ulimit or /etc/security/limits.conf
methods.

On Wed, Oct 6, 2010 at 1:51 PM, Rana Aich <aichrana@gmail.com> wrote:
> Hello Experts,
> I see a queer behavior from on of the Cassandra nodes in my cluster where
> the data is not flushed off Commitlogs and the Commitlog file grows in
> number. I was inserting the data into the cluster and since yesterday this
> node had more than 900 commitlog files.
> -rw-r--r-- 1 dev dev  134218072 2010-10-05 15:11 CommitLog-1286218976629.log
> -rw-r--r-- 1 dev dev  134217909 2010-10-05 15:11 CommitLog-1286316625590.log
> -rw-r--r-- 1 dev dev  134217872 2010-10-05 15:15 CommitLog-1286316673050.log
> -rw-r--r-- 1 dev dev  134217957 2010-10-05 15:19 CommitLog-1286316917618.log
> -rw-r--r-- 1 dev dev  134217764 2010-10-05 15:21 CommitLog-1286317178922.log
> -rw-r--r-- 1 dev dev  134217871 2010-10-05 15:24 CommitLog-1286317311372.log
> -rw-r--r-- 1 dev dev  134218029 2010-10-05 15:25 CommitLog-1286317446787.log
> -rw-r--r-- 1 dev dev  134218065 2010-10-05 15:26 CommitLog-1286317506892.log
> -rw-r--r-- 1 dev dev  134218100 2010-10-05 15:27 CommitLog-1286317567927.log
> -rw-r--r-- 1 dev dev  134218020 2010-10-05 15:29 CommitLog-1286317634668.log
> -rw-r--r-- 1 dev dev  134217922 2010-10-05 15:31 CommitLog-1286317741576.log
> -rw-r--r-- 1 dev dev  134217944 2010-10-05 15:32 CommitLog-1286317887655.log
> -rw-r--r-- 1 dev dev  134217961 2010-10-05 15:33 CommitLog-1286317968395.log
> -rw-r--r-- 1 dev dev  134217926 2010-10-05 15:34 CommitLog-1286318033241.log
> -rw-r--r-- 1 dev dev  134218045 2010-10-05 15:36 CommitLog-1286318097727.log
> -rw-r--r-- 1 dev dev  134218027 2010-10-05 15:37 CommitLog-1286318190654.log
> -rw-r--r-- 1 dev dev  134218060 2010-10-05 15:38 CommitLog-1286318277987.log
> -rw-r--r-- 1 dev dev  134218055 2010-10-05 15:41 CommitLog-1286318338561.log
> -rw-r--r-- 1 dev dev  134218069 2010-10-05 15:41 CommitLog-1286318461180.log
> -rw-r--r-- 1 dev dev  134217791 2010-10-05 15:43 CommitLog-1286318519193.log
> -rw-r--r-- 1 dev dev  134218066 2010-10-05 15:44 CommitLog-1286318609960.log
> -rw-r--r-- 1 dev dev  134218093 2010-10-05 15:45 CommitLog-1286318692656.log
> It seems that commitlog files spawned every minute and when it grew over 900
> files the Cassandra node went down.
> Now I'm trying to start my node but it has started to read all of those
> Commitlog files and takes huge amount of time to restart.
> Can anyone throw some light on this unusual behavior. Till yesterday before
> starting the data insertion the node had almost 550GB of data. Incidentally
> I did stop another node prior to inserting the data because it was 100%
> full. I'm using RandomPartitioner with Initial Tokens.
> Thanking all in advance.
> Rana
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message