Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <4E8DCEE9.2080105@unitedgames.com>
Date: Thu, 06 Oct 2011 17:53:13 +0200
From: Stefan Reek <stefan@unitedgames.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.1.16) Gecko/20110307 Iceowl/1.0b1 Icedove/3.0.11
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Re: Batch writes getting slow
References: <4E8DC49B.9040103@unitedgames.com>
 <CALdd-zg8T0pmU+rV1jfyoQQoDcSSKuapkzX+36zS5b5KDKRzbA@mail.gmail.com>
In-Reply-To: 
 <CALdd-zg8T0pmU+rV1jfyoQQoDcSSKuapkzX+36zS5b5KDKRzbA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 10/06/2011 05:26 PM, Jonathan Ellis wrote:
> On Thu, Oct 6, 2011 at 10:09 AM, Stefan Reek<stefan@unitedgames.com>  wrote:
>    
>> I can see that during the times the writing gets slow there are ~3000
>> pending tasks, but they disappear quickly.
>>      
> Your best bet is to make the write load more constant and less bursty.
>   If you really do need to handle bursts like that with low latency,
> then you probably do need more hardware.  (But make sure you've
> covered the basics first, like putting commitlog on a separate
> device.)
>
>    
We really need to make sure that all writes are succesfully written 
before the next batch has to be written,
so the bursts are unavoidable I think.
We do have the commitlogs on separate devices, are there any other 
basics that I could have forgotten, or
any parameters that are important for write performance? As I understand 
it the flush thresholds mainly
influence read performance instead of write performance.

Would it make any difference to write the data with more threads from 
the client, as that's something we can easily tune.

>> I can also see that Cassandra gradually takes more and more memory,
>> eventually filling up the 16GB that is assigned to it, although it
>> doesn't go out of memory.
>> Is this normal behaviour? I expected to see more of a sawtooth...
>>      
> Yes, that is normal from the OS's view of the JVM.  If you want to see
> the sawtooth you'd need to look at the JVM's internal metrics, e.g.,
> with jconsole.
>
>    
I can see the sawtooth in the JVM only for Par Eden and Par Survivor 
space, the CMS Old Gen space just keeps on growing though.


Anyway, thanks for the quick reply.

Regards,

Stefan