cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Cassandra startup times
Date Fri, 20 Jul 2012 14:37:24 GMT
Local storage is more the just the norm. Unless you have a very good
reason you should not be using NFS.

Edward

On Fri, Jul 20, 2012 at 4:55 AM, aaron morton <aaron@thelastpickle.com> wrote:
> 45 minutes for 90GB is high.
>
> The odd one out here is using NFS, local storage is the norm.
>
> I would look into the NFS first, low network IO and low CPU would suggest it
> is waiting on disk IO. The simple thing would be to try starting from local
> disk and see how much faster it is. Or look at the await time in iostat.
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 19/07/2012, at 4:54 PM, Ben Kaehne wrote:
>
> Good evening,
>
> I am interested in improving the startup time of our cassandra cluster.
>
> We have a 3 node cluster (replication factor of 3) in which our application
> requires quorum reads and writes to function.
>
> Each machine is well specced with 24gig of ram, 10 cores, jna enabled etc.
>
> On each server our keyspace files are so far around 90 Gb (stored on NFS
> although I am not seeing signs that we have much network io). This size will
> grow in future.
>
> Our startup time for 1 server at the moment is greater then half an hour (45
> minutes to 50 minutes even) which is putting a risk factor on the resiliance
> of our service. I have tried version 1.09 to latest 1.12.
>
> I do not see too much system utilization while starting either.
>
> I gazed apon an article suggesting increased speed in 1.2 although when I
> set it up, it did not seem to be any faster at all (if not slower).
>
> I was observing what was happening during startup and I noticed (via
> strace), cassandra was doing lots of 8 byte reads from:
>
>
> /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1871-CompressionInfo.db
>
> /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1874-CompressionInfo.db
>
> Also... Is there someone I can change the 8 byte reads to something greater?
> 8 byte reads across NFS is terribly inefficient (and I am guessing the cause
> of our terribly slow startup times).
>
> Regards,
>
> --
> -Ben
>
>

Mime
View raw message