Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: 74.125.83.44 is neither permitted nor
 denied by domain of nathan@milford.io)
MIME-Version: 1.0
From: Nathan Milford <nathan@milford.io>
Date: Thu, 14 Apr 2011 12:34:59 -0400
Message-ID: <BANLkTi=1tQjJXgMzBKXrnicvCqf8jF3jYA@mail.gmail.com>
Subject: Stress testing disk configurations. Your thoughts?
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=90e6ba61379ec93f7a04a0e38213

--90e6ba61379ec93f7a04a0e38213
Content-Type: text/plain; charset=UTF-8

Ahoy,

I'm building out a new 0.7.4 cluster to migrate our 0.6.6 cluster to.

While I'm waiting for the dev-side to get time to work on their side of the
project I have a 10 node cluster evenly split across two data centers (NY &
LA) and was looking to do some testing while I could.

My primary focus is on disk configurations.  Space isn't a huge issue, our
current data set is ~30G on each node and I imagine that'll go up since I
intend on tweaking the RF on the new cluster.

Each node has 6 x 146G 10K SAS drives.  I want to test:

1) 6 disks in R0 where everything is written to the same stripe
2) 1 disk for OS+Commitlog and 5 disks in R0 for data.
3) 1 disk for OS+Commitlog and 5 individual disks defined
as separate data_file_directories.

I suspect I'll see best performance with option 3, but the issue has become
political\religious and there are internal doubts that separating the commit
log and data will truly improve performance despite documentation and logic
indicating otherwise.  Thus the test :)

Right now I've been tinkering and not being very scientific while I work out
a testing methodology and get used to the tools.  I've just been running
zznate's cassandra-stress against a single node and measuring the time it
takes to read and write N rows.

Unscientifically I've found that they all perform about the same. It is hard
to judge because, when writing to a single node, reads take exponentially
longer.  Writing 10M rows may take ~500 seconds, but reading will take ~5000
seconds.  I'm sure this will even out when I test across more than one node.

Early next week I'll be able to test against all 10 nodes with a realistic
replication factor.

I'd really love to hear some people's thoughts on methodologies and what I
should be looking at/for other than iostat and the time for the test to
inset/read.

Thanks,
nathan

--90e6ba61379ec93f7a04a0e38213
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div>Ahoy,</div><div><br></div><div>I&#39;m building out a new 0.7.4 cluste=
r to migrate our 0.6.6 cluster to.</div><div><br></div><div>While I&#39;m w=
aiting for the dev-side to get time to work on=C2=A0their=C2=A0side of the =
project I have a 10 node cluster evenly split across two data centers (NY &=
amp; LA) and was looking to do some testing while I could.</div>

<div><br></div><div>My primary focus is on disk configurations. =C2=A0Space=
 isn&#39;t a huge issue, our current data set is ~30G on each node and I im=
agine that&#39;ll go up since I intend on tweaking the RF on the new cluste=
r.</div>

<div><br></div><div>Each node has 6 x 146G 10K SAS drives. =C2=A0I want to =
test:</div><div><br></div><div>1) 6 disks in R0 where everything is written=
 to the same stripe</div><div>2) 1 disk for OS+Commitlog and 5 disks in R0 =
for data.</div>

<div>3) 1 disk for OS+Commitlog and 5 individual disks defined as=C2=A0sepa=
rate=C2=A0data_file_directories.</div><div><br></div><div>I suspect I&#39;l=
l see best performance with option 3, but the issue has become political\re=
ligious and there are internal doubts that=C2=A0separating=C2=A0the commit =
log and data will truly improve performance despite documentation and logic=
 indicating otherwise. =C2=A0Thus the test :)</div>

<div><br></div><div>Right now I&#39;ve been tinkering and not being very sc=
ientific while I work out a testing methodology and get used to the tools. =
=C2=A0I&#39;ve just been running zznate&#39;s cassandra-stress against a si=
ngle node and measuring the time it takes to read and write N rows.=C2=A0</=
div>

<div><br></div><div>Unscientifically I&#39;ve found that they all perform a=
bout the same. It is hard to judge because, when writing to a single node, =
reads take exponentially longer. =C2=A0Writing 10M rows may take ~500 secon=
ds, but reading will take ~5000 seconds. =C2=A0I&#39;m sure this will even =
out when I test across more than one node.</div>

<div><br></div><div>Early next week I&#39;ll be able to test against all 10=
 nodes with a realistic replication factor.</div><div><br></div><div>I&#39;=
d really love to hear some people&#39;s thoughts on methodologies and what =
I should be looking at/for other than iostat and the time for the test to i=
nset/read.</div>

<div><br></div><div>Thanks,</div><div>nathan</div><div><br></div>

--90e6ba61379ec93f7a04a0e38213--