Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
Date: Thu, 19 Jul 2012 14:54:14 +1000
Message-ID: 
 <CAGYBkn-LCRAnBDmX2n_3oF+41rogmLSg-QGLxD++w81qDmuC2g@mail.gmail.com>
Subject: Cassandra startup times
From: Ben Kaehne <ben.kaehne@sirca.org.au>
To: user@cassandra.apache.org
Cc: Franc Carter <franc.carter@sirca.org.au>,
 David Nelson <david.nelson@sirca.org.au>,
	Andrew Thomson <andrew.thomson@sirca.org.au>
Content-Type: multipart/alternative; boundary=20cf3071d0b239dbdf04c52792d4

--20cf3071d0b239dbdf04c52792d4
Content-Type: text/plain; charset=ISO-8859-1

Good evening,

I am interested in improving the startup time of our cassandra cluster.

We have a 3 node cluster (replication factor of 3) in which our application
requires quorum reads and writes to function.

Each machine is well specced with 24gig of ram, 10 cores, jna enabled etc.

On each server our keyspace files are so far around 90 Gb (stored on NFS
although I am not seeing signs that we have much network io). This size
will grow in future.

Our startup time for 1 server at the moment is greater then half an hour
(45 minutes to 50 minutes even) which is putting a risk factor on the
resiliance of our service. I have tried version 1.09 to latest 1.12.

I do not see too much system utilization while starting either.

I gazed apon an article suggesting increased speed in 1.2 although when I
set it up, it did not seem to be any faster at all (if not slower).

I was observing what was happening during startup and I noticed (via
strace), cassandra was doing lots of 8 byte reads from:

 /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1871-CompressionInfo.db
 /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1874-CompressionInfo.db

Also... Is there someone I can change the 8 byte reads to something
greater? 8 byte reads across NFS is terribly inefficient (and I am guessing
the cause of our terribly slow startup times).

Regards,

-- 
-Ben

--20cf3071d0b239dbdf04c52792d4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Good evening,<div><br></div><div>I am interested in improving the startup t=
ime of our cassandra cluster.</div><div><br></div><div>We have a 3 node clu=
ster (replication factor of 3) in which our application requires quorum rea=
ds and writes to function.</div>
<div><br></div><div>Each machine is well specced with 24gig of ram, 10 core=
s, jna enabled etc.</div><div><br></div><div>On each server our keyspace fi=
les are so far around 90 Gb (stored on NFS although I am not seeing signs t=
hat we have much network io). This size will grow in future.</div>
<div><div><div><br></div><div>Our startup time for 1 server at the moment i=
s greater then half an hour (45 minutes to 50 minutes even) which is puttin=
g a risk factor on the resiliance of our service. I have tried version 1.09=
 to latest 1.12.</div>
<div><br></div><div>I do not see too much system=A0utilization=A0while star=
ting either.</div><div><br></div><div>I gazed apon an article suggesting in=
creased speed in 1.2 although when I set it up, it did not seem to be any f=
aster at all (if not slower).</div>
<div><br></div><div>I was observing what was happening during startup and I=
 noticed (via strace), cassandra was doing lots of 8 byte reads from:</div>=
<div><br></div><div>=A0/var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYY=
Y-hc-1871-CompressionInfo.db</div>
<div><div>=A0/var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1874-=
CompressionInfo.db</div></div><div><br></div><div>Also... Is there someone =
I can change the 8 byte reads to something greater? 8 byte reads across NFS=
 is terribly inefficient (and I am guessing the cause of our terribly slow =
startup times).</div>
<div><br></div><div>Regards,</div><div><br></div>-- <br>-Ben<br>
</div></div>

--20cf3071d0b239dbdf04c52792d4--