Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of tsaloranta@gmail.com designates
 209.85.216.196 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=R0d9Ot+fA0lia1wzhj2re/LqOB1ByVthx/qfxDV7sq2fPssiglxx8WYQPaUzMb5370
         hvnoOHom5FQuxPG+xDqQOW18/hFsSAUI5FFlSeAJsiac7kG4AU9vNwtHxmCumz8SaiSq
         hJliEcUvb2w1NyHsjpiXgSLFHN6BNRrh6BQhw=
MIME-Version: 1.0
In-Reply-To: <a4f278e51002160625i40f58ac6v9d2ad532bf108fae@mail.gmail.com>
References: <a4f278e51002160625i40f58ac6v9d2ad532bf108fae@mail.gmail.com>
Date: Tue, 16 Feb 2010 10:28:37 -0800
Message-ID: <5f7770581002161028s21c54cdfk846e2c973d06aaf6@mail.gmail.com>
Subject: Re: cassandra freezes
From: Tatu Saloranta <tsaloranta@gmail.com>
To: cassandra-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Feb 16, 2010 at 6:25 AM, Boris Shulman <shulmanb@gmail.com> wrote:
> Hello, I'm running some benchmarks on 2 cassandra nodes each running
> on 8 cores machine with 16G RAM, 10G for Java heap. I've noticed that
> during benchmarks with numerous writes cassandra just freeze for
> several minutes (in those benchmarks I'm writing batches of 10 columns
> with 1K data each for every key in a single CF). Usually after
> performing 50K writes I'm getting a TimeOutException and cassandra
> just freezes. What configuration changes can I make in order to
> prevent this? Is it possible that my setup just can't handle the load?
> How can I calculate the number of casandra nodes for a desired load?

One thing that can cause seeming lockups is garbage collector. So
enabling GC debug output would be heplful, to see GC activity. Some
collector (CMS specifically) can stop the system for very long time,
up to minutes. This is not necessarily the root cause, but is easy to
rule out.
Beyond this, getting a stack trace during lockup would make sense.
That can pinpoint what threads are doing, or what they are blocked on
in case there is a deadlock or heavy contention on some shared
resource.

-+ Tatu +-