Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of rohit.kommareddy@gmail.com
 designates 209.85.212.44 as permitted sender)
MIME-Version: 1.0
Date: Fri, 14 Sep 2012 18:54:05 +0530
Message-ID: 
 <CAE1RT7d4BKGc2YBqHvdr-uCwdSi2sjt8sTb+EkCdtmGA1+8LhA@mail.gmail.com>
Subject: Cassandra node going down
From: rohit reddy <rohit.kommareddy@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=f46d0432b1c28bbd5004c9a95686

--f46d0432b1c28bbd5004c9a95686
Content-Type: text/plain; charset=UTF-8

Hi,

I'm facing a problem in Cassandra cluster deployed on EC2 where the node is
going down under write load.

I have configured a cluster of 4 Large EC2 nodes with RF of 2.
All nodes are instance storage backed. DISK is RAID0 with 800GB

I'm pumping in write requests at about 4000 writes/sec. One of the node
went down under this load. The total data size in each node was not more
than 7GB
Got the following WARN messages in the LOG file...

1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601
2. Heap is 0.7515559786053904 full.  You may need to reduce memtable and/or
cache sizes.  Cassandra will now flush up to the two largest memtables to
free up memory.  Adjust flush_largest_memtables_at threshold in
cassandra.yaml if you don't want Cassandra to do
this automatically
3. WARN [CompactionExecutor:570] 2012-09-14 11:45:12,024
CompactionTask.java (line 84) insufficient space to compact all requested
files

All cassandra settings are default settings.
Do i need to tune anything to support this write rate?

Thanks
Rohit

--f46d0432b1c28bbd5004c9a95686
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div>Hi,</div><div><br></div><div>I&#39;m facing a problem in Cassandra clu=
ster deployed on EC2 where the node is going down under write load.</div><d=
iv><br></div><div>I have configured a cluster of 4 Large EC2 nodes with RF =
of 2.</div>
<div>All nodes are instance storage backed. DISK is RAID0 with 800GB</div><=
div><br></div><div>I&#39;m pumping in write requests at about 4000 writes/s=
ec. One of the node went down under this load. The total data size in each =
node was not more than 7GB</div>
<div>Got the following WARN messages in the LOG file...</div><div><br></div=
><div>1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601=
</div><div>2. Heap is 0.7515559786053904 full. =C2=A0You may need to reduce=
 memtable and/or cache sizes. =C2=A0Cassandra will now flush up to the two =
largest memtables to free up memory. =C2=A0Adjust flush_largest_memtables_a=
t threshold in cassandra.yaml if you don&#39;t want Cassandra to do</div>
<div>this automatically</div><div>3. WARN [CompactionExecutor:570] 2012-09-=
14 11:45:12,024 CompactionTask.java (line 84) insufficient space to compact=
 all requested files</div><div><br></div><div>All cassandra settings are de=
fault settings.</div>
<div>Do i need to tune anything to support this write rate?</div><div><br><=
/div><div>Thanks</div><div>Rohit</div><div><br></div>

--f46d0432b1c28bbd5004c9a95686--