cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra startup times
Date Fri, 20 Jul 2012 08:55:04 GMT
45 minutes for 90GB is high. 

The odd one out here is using NFS, local storage is the norm. 

I would look into the NFS first, low network IO and low CPU would suggest it is waiting on
disk IO. The simple thing would be to try starting from local disk and see how much faster
it is. Or look at the await time in iostat.

Hope that helps.

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 19/07/2012, at 4:54 PM, Ben Kaehne wrote:

> Good evening,
> 
> I am interested in improving the startup time of our cassandra cluster.
> 
> We have a 3 node cluster (replication factor of 3) in which our application requires
quorum reads and writes to function.
> 
> Each machine is well specced with 24gig of ram, 10 cores, jna enabled etc.
> 
> On each server our keyspace files are so far around 90 Gb (stored on NFS although I am
not seeing signs that we have much network io). This size will grow in future.
> 
> Our startup time for 1 server at the moment is greater then half an hour (45 minutes
to 50 minutes even) which is putting a risk factor on the resiliance of our service. I have
tried version 1.09 to latest 1.12.
> 
> I do not see too much system utilization while starting either.
> 
> I gazed apon an article suggesting increased speed in 1.2 although when I set it up,
it did not seem to be any faster at all (if not slower).
> 
> I was observing what was happening during startup and I noticed (via strace), cassandra
was doing lots of 8 byte reads from:
> 
>  /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1871-CompressionInfo.db
>  /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1874-CompressionInfo.db
> 
> Also... Is there someone I can change the 8 byte reads to something greater? 8 byte reads
across NFS is terribly inefficient (and I am guessing the cause of our terribly slow startup
times).
> 
> Regards,
> 
> -- 
> -Ben


Mime
View raw message