Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
Sender: les.hazlewood@katasoft.com
From: Les Hazlewood <les@katasoft.com>
Date: Wed, 22 Jun 2011 14:24:13 -0700
Message-ID: <BANLkTimqkZgp36uH9q98PSXm9CjjwX__uA@mail.gmail.com>
Subject: 99.999% uptime - Operations Best Practices?
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=bcaec52be44d3ba41b04a653984e

--bcaec52be44d3ba41b04a653984e
Content-Type: text/plain; charset=ISO-8859-1

I'm planning on using Cassandra as a product's core data store, and it is
imperative that it never goes down or loses data, even in the event of a
data center failure.  This uptime requirement ("five nines": 99.999% uptime)
w/ WAN capabilities is largely what led me to choose Cassandra over other
NoSQL products, given its history and 'from the ground up' design for such
operational benefits.

However, in a recent thread, a user indicated that all 4 of 4 of his
Cassandra instances were down because the OS killed the Java processes due
to memory starvation, and all 4 instances went down in a relatively short
period of time of each other.  Another user helped out and replied that
running 0.8 and nodetool repair on each node regularly via a cron job (once
a day?) seems to work for him.

Naturally this was disconcerting to read, given our needs for a Highly
Available product - we'd be royally screwed if this ever happened to us.
 But given Cassandra's history and it's current production use, I'm aware
that this HA/uptime is being achieved today, and I believe it is certainly
achievable.

So, is there a collective set of guidelines or best practices to ensure this
problem (or unavailability due to OOM) can be easily managed?

Things like memory settings, initial GC recommendations, cron
recommendations, ulimit settings, etc. that can be bundled up as a
best-practices "Production Kickstart"?

Could anyone share their nuggets of wisdom or point me to resources where
this may already exist?

Thanks!

Best regards,

Les

--bcaec52be44d3ba41b04a653984e
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I&#39;m planning on using Cassandra as a product&#39;s core data store, and=
 it is imperative that it never goes down or loses data, even in the event =
of a data center failure. =A0This uptime requirement (&quot;five nines&quot=
;: 99.999% uptime) w/ WAN capabilities is largely what led me to choose Cas=
sandra over other NoSQL products, given its history and &#39;from the groun=
d up&#39; design for such operational benefits.<div>


<br></div><div>However, in a recent thread, a user indicated that all 4 of =
4 of his Cassandra instances were down because the OS killed the Java proce=
sses due to memory starvation, and all 4 instances went down in a relativel=
y short period of time of each other. =A0Another user helped out and replie=
d that running 0.8 and nodetool repair on each node regularly via a cron jo=
b (once a day?) seems to work for him.</div>

<div><br></div><div>Naturally this was disconcerting to read, given our nee=
ds for a Highly Available product - we&#39;d be royally screwed if this eve=
r happened to us. =A0But given Cassandra&#39;s history and it&#39;s current=
 production use, I&#39;m aware that this HA/uptime is being achieved today,=
 and I believe it is certainly achievable.</div>

<div><br></div><div>So, is there a collective set of guidelines or best pra=
ctices to ensure this problem (or unavailability due to OOM) can be easily =
managed?</div><div><br></div><div>Things like memory settings, initial GC r=
ecommendations, cron recommendations, ulimit settings, etc. that can be bun=
dled up as a best-practices &quot;Production Kickstart&quot;?</div>

<div><br></div><div>Could anyone share their nuggets of wisdom or point me =
to resources where this may already exist?</div><div><br>Thanks!</div><div>=
<br></div><div>Best regards,</div><div><div><div><br></div><div><span class=
=3D"Apple-style-span" style=3D"border-collapse: collapse; font-family: aria=
l, sans-serif; font-size: 13px; ">Les</span><br>


</div></div></div>

--bcaec52be44d3ba41b04a653984e--