incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Downing <tdown...@proteus-technologies.com>
Subject Re: Minimizing the impact of compaction on latency and throughput
Date Tue, 13 Jul 2010 09:19:08 GMT
On a related note:  I am running some feasibility tests looking for
high ingest rate capabilities.  While testing Cassandra the problem
I've encountered is that it runs out of file handles during compaction.
Until that event, there was no significant impact on throughput as
I was using it (0.1 query per second, ~10,000 records/query).

Up to this point, Cassandra was definitely in the lead among the
alternatives.

This was with 0.6.3, single node installation.  Ingest rate was about
4000 records/second, 1600 bytes/record, 24 bytes/key, using
batch_mutate.  Unfortunately, Cassandra seems unable to recover from
this state.  This occurs at about 100M records in the database.

I tried a 0.7.0 snapshot, but encountered earlier and worse problems.

The machine is 4 CPU AMD64 2.2GHz, 4GB.  There was no swapping.

The only mention of running out of file handles I found in the archives
or the defect list was related to queries - but I am notoriously blind.
I see the same behavior running ingest only, no queries

I've blown away the logs and data, but if there is interest in further info
on this problem, such as stacktrace and specific numbers, I will re-run
the test and send them along.

Thanks much for all your work

Thomas Downing

On 7/12/2010 10:52 PM, Jonathan Ellis wrote:
> This looks relevant:
> http://chbits.blogspot.com/2010/06/lucene-and-fadvisemadvise.html (see
> comments for directions to code sample)
>
> On Fri, Jul 9, 2010 at 1:52 AM, Peter Schuller
> <peter.schuller@infidyne.com>  wrote:
>    
>>> It might be worth experimenting with posix_fadvise.  I don't think
>>> implementing our own i/o scheduler or rate-limiter would be as good a
>>> use of time (it sounds like you're on that page too).
>>>        
>> Ok. And yes I mostly agree, although I can imagine circumstances where
>> a pretty simple rate limiter might help significantly - albeit be
>> something that has to be tweaked very specifically for the
>> situation/hardware rather than being auto-tuned.
>>
>> If I have the time I may look into posix_fadvise() to begin with (but
>> I'm not promising anything).
>>
>> Thanks for the input!
>>
>> --
>> / Peter Schuller
>>
>>      
>
>    

Mime
View raw message