Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 209.85.160.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=EXmiKW39ScpC2j2x+rYXC6vMEjsxHVyh6oIzc21OfZ2INR+2O6iGUTN3FHLvLevZwT
         8u5X8YlCL27iyDf+wYmPeq+LVn8wQB8KgOcAxhCEcUaEGRNVO26QcGVN7cKg4XCPyHsr
         5FCi8Qh+rCFIXpNt2TnGTQ3xGVwWhU87XDb7M=
MIME-Version: 1.0
In-Reply-To: <AANLkTiledwIsiiWvwX6sAEnsy3XkrlVucVDbQgW9-7wf@mail.gmail.com>
References: <7A93E9E4-BC15-479C-A4CD-0E96CEFB540E@clearspring.com>
	<AANLkTikYs8LcZS4wXX2aMLRbXkNvmWOXFFePQNXbQc6b@mail.gmail.com>
	<AANLkTiledwIsiiWvwX6sAEnsy3XkrlVucVDbQgW9-7wf@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Wed, 30 Jun 2010 15:59:39 -0700
Message-ID: <AANLkTikFCd9qMQ3KoW6ZMmgbCIoQ8Q1-ZDk8B3dEh3gI@mail.gmail.com>
Subject: Re: Cluster-wide pause
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

use a larger heap or throttle your writes

On Wed, Jun 30, 2010 at 12:12 PM, Steve Lihn <stevelihn@gmail.com> wrote:
> Jon,
> I am experimenting writing 8 million rows into Cassandra and also
> experienced some random timeouts, even with 10-second timeout parameter.
> How do I avoid such timeout at all cost? (I.e. At this time, my priority =
is
> to finish the end-to-end test. Don't want the program to fail at all.)
> This is a one-node server with ms1G and mx2GB. The rest in cassandra.in.s=
h
> are default I think.
>
> Steve
>
> On Wed, Jun 30, 2010 at 1:58 AM, Jonathan Ellis <jbellis@gmail.com> wrote=
:
>>
>> You could be seeing GC pauses. Did you increase the heap size you gave
>> Cassandra, when you increased your VM size?
>>
>> On Tue, Jun 29, 2010 at 11:57 AM, Oren Benjamin <oren@clearspring.com>
>> wrote:
>> > Hi all - first timer here.
>> >
>> > I'm experimenting with Cassandra on Rackspace Cloud. =A0Started with 4=
GB
>> > nodes and saw read latency spikes while streaming was taking place, so=
 I
>> > increased to 8GB to see if limited memory was the issue. =A0Now I'm se=
eing
>> > very strange behavior during any period that writes are taking place. =
=A0The
>> > entire (6 node) cluster seems to pause for periods of as much as 5-8 s=
ec.
>> > =A0By that I mean all the stats (cpu, disk, network IO monitored via d=
stat)
>> > drop to zero or near zero on all nodes simultaneously. =A0Does anyone =
have
>> > experience with Cassandra on Rackspace or any idea what's going on her=
e?
>> >
>> > The pauses are short enough that it's difficult to introspect the
>> > application and determine what it's doing during the pause, but long e=
nough
>> > to cause unacceptable latency for any service built on top of it.
>> >
>> > Any ideas or debugging methods would be greatly appreciated,
>> >
>> > =A0-- Oren
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com