Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 74.125.82.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <4E8EBCBA.1020807@unitedgames.com>
References: <4E8DC49B.9040103@unitedgames.com>
 <CALdd-zg8T0pmU+rV1jfyoQQoDcSSKuapkzX+36zS5b5KDKRzbA@mail.gmail.com>
 <4E8DCEE9.2080105@unitedgames.com>
 <CALdd-zgJ0ynDxF_keSZymyoq0o=XB0Gkm+=Ozq9DEVHDWwSbag@mail.gmail.com>
 <4E8EBCBA.1020807@unitedgames.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Fri, 7 Oct 2011 08:06:15 -0500
Message-ID: 
 <CALdd-ziX-p97EXGWi6EatnofAnEq9b9_oBma4x8RRJ+STOMnnA@mail.gmail.com>
Subject: Re: Batch writes getting slow
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Look at the OOM section in
http://www.datastax.com/docs/0.6/troubleshooting/index

On Fri, Oct 7, 2011 at 3:47 AM, Stefan Reek <stefan@unitedgames.com> wrote:
> Is it actually filling up enough to trigger an old-gen CMS gc?
>
>
> Yes, it fills up to the 16G and then it starts doing the CMS gc's which
> dramatically decreases the performance.
> I'm still not sure why it does this, as a nodetool info states the load a=
s
> less than 4G.
> Any ideas?
>
>
> On 10/06/2011 06:15 PM, Jonathan Ellis wrote:
>>
>> On Thu, Oct 6, 2011 at 10:53 AM, Stefan Reek<stefan@unitedgames.com>
>> =A0wrote:
>>
>>>
>>> We do have the commitlogs on separate devices, are there any other basi=
cs
>>> that I could have forgotten, or
>>> any parameters that are important for write performance?
>>>
>>
>> 1.0 write performance is something like 30% better... =A0I don't think
>> there's anything else you'll find for "free."
>>
>>
>>>
>>> As I understand it
>>> the flush thresholds mainly
>>> influence read performance instead of write performance.
>>>
>>
>> It can affect write performance if you're flushing really small
>> sstables, but I doubt that's the problem here.
>>
>>
>>>
>>> Would it make any difference to write the data with more threads from t=
he
>>> client, as that's something we can easily tune.
>>>
>>
>> Not in this case because Cassandra turns the batch into single-row
>> writes internally, so it gets parallelized that way.
>>
>> If you can avoid waiting for One Big Batch and stream changes in as
>> they happen, that would help.
>>
>>
>>>
>>> I can see the sawtooth in the JVM only for Par Eden and Par Survivor
>>> space,
>>> the CMS Old Gen space just keeps on growing though.
>>>
>>
>> Is it actually filling up enough to trigger an old-gen CMS gc?
>>
>>
>
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com