Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <BANLkTi=z3EF7m4QXD=wBjJNNNA9cWid70A@mail.gmail.com>
References: <BANLkTimqkZgp36uH9q98PSXm9CjjwX__uA@mail.gmail.com>
	<BANLkTinbph4KFKtRWNFirJkm+hZpFnXTLg@mail.gmail.com>
	<4E026E4C.40601@gmail.com>
	<BANLkTi=Avnz=1N7VwaoZ_jnqJvk53sMW4g@mail.gmail.com>
	<E6714820-DF98-4CD5-AF9C-8F8AF5E1506A@gmail.com>
	<BANLkTim6j2higfQJnv_vPtPak1b0prPYNg@mail.gmail.com>
	<BANLkTi=q26M9FBDTH6DE6ykcQT2ceQ4rCA@mail.gmail.com>
	<BANLkTi=z3EF7m4QXD=wBjJNNNA9cWid70A@mail.gmail.com>
Date: Thu, 23 Jun 2011 10:43:40 +0300
Message-ID: <BANLkTinKCmMXcOkzeid9nhzrsLr5rJ+AaQ@mail.gmail.com>
Subject: Re: 99.999% uptime - Operations Best Practices?
From: David Boxenhorn <david@citypath.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I think very high uptime, and very low data loss is achievable in
Cassandra, but, for new users there are TONS of gotchas. You really
have to know what you're doing, and I doubt that many people acquire
that knowledge without making a lot of mistakes.

I see above that most people are talking about configuration issues.
But, the first thing that you will probably do, before you have any
experience with Cassandra(!), is architect your system. Architecture
is not easily changed when you bump into a gotcha, and for some reason
you really have to search the literature well to find out about them.
So, my contributions:

The too many CFs problem. Cassandra doesn't do well with many column
families. If you come from a relational world, a real application can
easily have hundreds of tables. Even if you combine them into entities
(which is the Cassandra way), you can easily end up with dozens of
entities. The most natural thing for someone with a relational
background is have one CF per entity, plus indexes according to your
needs. Don't do it. You need to store multiple entities in the same
CF. Group them together according to access patterns (i.e. when you
use X,  you probably also need Y), and distinguish them by adding a
prefix to their keys (e.g. entityName@key).

Don't use supercolumns, use composite columns. Supercolumns are
disfavored by the Cassandra community and are slowly being orphaned.
For example, secondary indexes don't work on supercolumns. Nor does
CQL. Bugs crop up with supercolumns that don't happen with regular
columns because internally there's a huge separate code base for
supercolumns, and every new feature is designed and implemented for
regular columns and then retrofitted for supercolumns (or not).

There should really be a database of gotchas somewhere, and how they
were solved...

On Thu, Jun 23, 2011 at 6:57 AM, Les Hazlewood <les@katasoft.com> wrote:
> Edward,
> Thank you so much for this reply - this is great stuff, and I really
> appreciate it.
> You'll be happy to know that I've already pre-ordered your book. =A0I'm
> looking forward to it! (When is the ship date?)
> Best regards,
> Les
>
> On Wed, Jun 22, 2011 at 7:03 PM, Edward Capriolo <edlinuxguru@gmail.com>
> wrote:
>>
>>
>> On Wed, Jun 22, 2011 at 8:31 PM, Les Hazlewood <les@katasoft.com> wrote:
>>>
>>> Hi Thoku,
>>> You were able to more concisely represent my intentions (and their
>>> reasoning) in this thread than I was able to do so myself. =A0Thanks!
>>>
>>> On Wed, Jun 22, 2011 at 5:14 PM, Thoku Hansen <thokuh@gmail.com> wrote:
>>>>
>>>> I think that Les's question was reasonable. Why *not* ask the communit=
y
>>>> for the 'gotchas'?
>>>> Whether the info is already documented or not, it could be an
>>>> opportunity to improve the documentation based on users' perception.
>>>> The "you just have to learn" responses are fair also, but that reminds
>>>> me of the days when running Oracle was a black art, and accumulated wi=
sdom
>>>> made DBAs irreplaceable.
>>>
>>> Yes, this was my initial concern. =A0I know that Cassandra is still you=
ng,
>>> and I expect this to be the norm for a while, but I was hoping to make =
that
>>> process a bit easier (for me and anyone else reading this thread in the
>>> future).
>>>>
>>>> Some recommendations *are* documented, but they are dispersed / stale =
/
>>>> contradictory / or counter-intuitive.
>>>> Others have not been documented in the wiki nor in DataStax's doco, an=
d
>>>> are instead learned=A0anecdotally or=A0The Hard Way.
>>>> For example, whether documented or not, some of the 'gotchas' that I
>>>> encountered when I first started working with Cassandra were:
>>>> * Don't use OpenJDK. Prefer the Sun JDK. (Wiki says this,=A0Jira says
>>>> that).
>>>> * Its not viable to run without JNA installed.
>>>> * Disable swap memory.
>>>> * Need to run nodetool repair on a regular basis.
>>>> I'm looking forward to=A0Edward Capriolo's Cassandra book=A0which Les =
will
>>>> probably find helpful.
>>>
>>> Thanks for linking to this. =A0I'm pre-ordering right away.
>>> And thanks for the pointers, they are exactly the kind of enumerated
>>> things I was looking to elicit. =A0These are the kinds of things that a=
re hard
>>> to track down in a single place. =A0I think it'd be nice for the commun=
ity to
>>> contribute this stuff to a single page ('best practices', 'checklist',
>>> whatever you want to call it). =A0It would certainly make things easier=
 when
>>> getting started.
>>> Thanks again,
>>> Les
>>
>> Since I got a plug on the book I will chip in again to the thread :)
>>
>> Some things that were mentioned already:
>>
>> Install JNA absolutely (without JNA the snapshot command has to fork to
>> hard link the sstables, I have seen clients backoff from this). Also the
>> performance focused Cassandra devs always try to squeeze out performance=
 by
>> utilizing more native features.
>>
>> OpenJDK vs Sun. I agree, almost always try to do what 'most others' do i=
n
>> production, this way you get surprised less.
>>
>> Other stuff:
>>
>> RAID. You might want to go RAID 1+0 if you are aiming for uptime. RAID 0
>> has better performance, but if you lose a node your capacity is diminish=
ed,
>> rebuilding and rejoining a node involves more manpower more steps and mo=
re
>> chances for human error.
>>
>> Collect statistics on the normal system items CPU, disk (size and
>> utilization), memory. Then collect the JMX cassandra counters and unders=
tand
>> how they interact. For example record ReadCount and WriteCount per colum=
n
>> family, then use try to determine how this effects disk utilization. You=
 can
>> use this for capacity planning. Then try using a key/row cache. Evaluate
>> again. Check the hit rate graph for your new cache. How did this effect =
your
>> disk? You want to head off anything that can be a performance killer lik=
e
>> traffic patterns changing or data growing significantly.
>>
>> Do not be short on hardware. I do not want to say "overbuy" but if uptim=
e
>> is important have spares drives and servers and have room to grow.
>>
>> Balance that ring :)
>>
>> I have not read the original thread concerning the problem you mentioned=
.
>> One way to avoid OOM is large amounts of RAM :) On a more serious note m=
ost
>> OOM's are caused by setting caches or memtables too large. If the OOM wa=
s
>> caused by a software bug, the cassandra devs are on the ball and move fa=
st.
>> I still suggest not jumping into a release right away. I know its hard t=
o
>> live without counters or CQL since new things are super cool. But if you
>> want all those 9s your going to have to stay disciplined. Unless a relea=
se
>> has a fix for a problem you think you have, stay a minor or revision bac=
k,
>> or at least wait some time before upgrading to it, and do some internal
>> confidence testing before pulling the trigger on an update.
>>
>> Almost all usecases demand that repair be run regularly due to the natur=
e
>> of distributed deletes.
>>
>> Other good tips, subscribe to all the mailing lists, and hang out in the
>> IRC channels cassandra, cassandra-dev, cassandra-ops. You get an osmoses
>> learning effect and you learn to fix or head off issues you never had.
>