cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Les Hazlewood <...@katasoft.com>
Subject Re: 99.999% uptime - Operations Best Practices?
Date Thu, 23 Jun 2011 03:57:04 GMT
Edward,

Thank you so much for this reply - this is great stuff, and I really
appreciate it.

You'll be happy to know that I've already pre-ordered your book.  I'm
looking forward to it! (When is the ship date?)

Best regards,

Les

On Wed, Jun 22, 2011 at 7:03 PM, Edward Capriolo <edlinuxguru@gmail.com>wrote:

>
>
> On Wed, Jun 22, 2011 at 8:31 PM, Les Hazlewood <les@katasoft.com> wrote:
>
>> Hi Thoku,
>>
>> You were able to more concisely represent my intentions (and their
>> reasoning) in this thread than I was able to do so myself.  Thanks!
>>
>> On Wed, Jun 22, 2011 at 5:14 PM, Thoku Hansen <thokuh@gmail.com> wrote:
>>
>>> I think that Les's question was reasonable. Why *not* ask the community
>>> for the 'gotchas'?
>>>
>>> Whether the info is already documented or not, it could be an opportunity
>>> to improve the documentation based on users' perception.
>>>
>>> The "you just have to learn" responses are fair also, but that reminds me
>>> of the days when running Oracle was a black art, and accumulated wisdom made
>>> DBAs irreplaceable.
>>>
>>
>> Yes, this was my initial concern.  I know that Cassandra is still young,
>> and I expect this to be the norm for a while, but I was hoping to make that
>> process a bit easier (for me and anyone else reading this thread in the
>> future).
>>
>> Some recommendations *are* documented, but they are dispersed / stale /
>>> contradictory / or counter-intuitive.
>>>
>>> Others have not been documented in the wiki nor in DataStax's doco, and
>>> are instead learned anecdotally or The Hard Way.
>>>
>>> For example, whether documented or not, some of the 'gotchas' that I
>>> encountered when I first started working with Cassandra were:
>>>
>>> * Don't use OpenJDK. Prefer the Sun JDK. (Wiki says this<http://wiki.apache.org/cassandra/GettingStarted>
>>> , Jira says that <https://issues.apache.org/jira/browse/CASSANDRA-2441>
>>> ).
>>> * Its not viable to run without JNA installed.
>>> * Disable swap memory.
>>> * Need to run nodetool repair on a regular basis.
>>>
>>> I'm looking forward to Edward Capriolo's Cassandra book<https://www.packtpub.com/cassandra-apache-high-performance-cookbook/book>
which
>>> Les will probably find helpful.
>>>
>>
>> Thanks for linking to this.  I'm pre-ordering right away.
>>
>> And thanks for the pointers, they are exactly the kind of enumerated
>> things I was looking to elicit.  These are the kinds of things that are hard
>> to track down in a single place.  I think it'd be nice for the community to
>> contribute this stuff to a single page ('best practices', 'checklist',
>> whatever you want to call it).  It would certainly make things easier when
>> getting started.
>>
>> Thanks again,
>>
>> Les
>>
>
> Since I got a plug on the book I will chip in again to the thread :)
>
> Some things that were mentioned already:
>
> Install JNA absolutely (without JNA the snapshot command has to fork to
> hard link the sstables, I have seen clients backoff from this). Also the
> performance focused Cassandra devs always try to squeeze out performance by
> utilizing more native features.
>
> OpenJDK vs Sun. I agree, almost always try to do what 'most others' do in
> production, this way you get surprised less.
>
> Other stuff:
>
> RAID. You might want to go RAID 1+0 if you are aiming for uptime. RAID 0
> has better performance, but if you lose a node your capacity is diminished,
> rebuilding and rejoining a node involves more manpower more steps and more
> chances for human error.
>
> Collect statistics on the normal system items CPU, disk (size and
> utilization), memory. Then collect the JMX cassandra counters and understand
> how they interact. For example record ReadCount and WriteCount per column
> family, then use try to determine how this effects disk utilization. You can
> use this for capacity planning. Then try using a key/row cache. Evaluate
> again. Check the hit rate graph for your new cache. How did this effect your
> disk? You want to head off anything that can be a performance killer like
> traffic patterns changing or data growing significantly.
>
> Do not be short on hardware. I do not want to say "overbuy" but if uptime
> is important have spares drives and servers and have room to grow.
>
> Balance that ring :)
>
> I have not read the original thread concerning the problem you mentioned.
> One way to avoid OOM is large amounts of RAM :) On a more serious note most
> OOM's are caused by setting caches or memtables too large. If the OOM was
> caused by a software bug, the cassandra devs are on the ball and move fast.
> I still suggest not jumping into a release right away. I know its hard to
> live without counters or CQL since new things are super cool. But if you
> want all those 9s your going to have to stay disciplined. Unless a release
> has a fix for a problem you think you have, stay a minor or revision back,
> or at least wait some time before upgrading to it, and do some internal
> confidence testing before pulling the trigger on an update.
>
> Almost all usecases demand that repair be run regularly due to the nature
> of distributed deletes.
>
> Other good tips, subscribe to all the mailing lists, and hang out in the
> IRC channels cassandra, cassandra-dev, cassandra-ops. You get an osmoses
> learning effect and you learn to fix or head off issues you never had.
>

Mime
View raw message