incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piavlo <lolitus...@gmail.com>
Subject Re: cassandra read latency help
Date Fri, 18 May 2012 06:58:55 GMT
On 05/18/2012 08:49 AM, Viktor Jevdokimov wrote:
>
> Row cache is ok until keys are not heavily updated, otherwise it 
> frequently invalidates and pressures GC.
>
According to http://www.datastax.com/docs/1.0/operations/tuning
"As of Cassandra 1.0, column family row caches are stored in native 
memory by default (outside of the Java heap).
This results in both a smaller per-row memory footprint and reduced JVM 
heap requirements, which helps keep the heap size manageable for good 
JVM garbage collection performance."
AFAIU it's outside of the Java heap only if JNA is used.

Then I tried row cache for for a few CF (with cassandra 1.0.9), to my 
surprise it just killed reads latency , and made very high cpu usage, 
the row hit rate was ~20% and reads/writes ~50/50.
The CFs are compressed (does it matter? does the row cache keep rows 
compressed or not?)
AFAIU with JNA off heap cache stores the rows in serialized form, so 
where does the high cpu come from?

> The high latency is from your batch of 100 keys. Review your data 
> model to avoid such reads, if you need low latency.
>
> 500M rows on one node, or on the cluster? Reading 100 random rows at 
> total of 40KB data from a data set of 180GB uncompressed under 30ms is 
> not an easy task.
>
>
>
> Best regards / Pagarbiai
> *Viktor Jevdokimov*
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com <mailto:Viktor.Jevdokimov@adform.com>
> Phone: +370 5 212 3063, Fax +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> Follow us on Twitter: @adforminsider 
> <http://twitter.com/#%21/adforminsider>
> What is Adform: watch this short video <http://vimeo.com/adform/display>
> 	
> Adform News <http://www.adform.com>
>
>
> Disclaimer: The information contained in this message and attachments 
> is intended solely for the attention and use of the named addressee 
> and may be confidential. If you are not the intended recipient, you 
> are reminded that the information remains the property of the sender. 
> You must not use, disclose, distribute, copy, print or rely on this 
> e-mail. If you have received this message in error, please contact the 
> sender immediately and irrevocably delete this message and any copies.
>
> *From:*Gurpreet Singh [mailto:gurpreet.singh@gmail.com]
> *Sent:* Thursday, May 17, 2012 20:24
> *To:* user@cassandra.apache.org
> *Subject:* Re: cassandra read latency help
>
> Thanks Viktor for the advice.
>
> Right now, i just have 1 node that i am testing against and i am using 
> CL one.
>
> Are you suggesting that the page cache might be doing better than the 
> row cache?
> I am getting row cache hit of 0.66 right now.
>
> /G
>
> On Thu, May 17, 2012 at 12:26 AM, Viktor Jevdokimov 
> <Viktor.Jevdokimov@adform.com <mailto:Viktor.Jevdokimov@adform.com>> 
> wrote:
>
> > Gurpreet Singh wrote:
> > Any ideas on what could help here bring down the read latency even 
> more ?
>
> Avoid Cassandra forwarding request to other nodes:
> - Use consistency level ONE;
> - Create data model to do single request with single key, since 
> different keys may belong to different nodes and requires forwarding 
> requests to them;
> - Use smart client to calculate token for key and select appropriate 
> node (primary or replica) by token range;
> - Turn off Dynamic Snitch (it may forward request to other replica 
> even it has the data);
> - Have all or hot data in page cache (no HDD disk IO) or use SSD;
> - If you do regular updates to key, do not use row cache, otherwise 
> you may try.
>
>
>
>
> Best regards / Pagarbiai
>
> Viktor Jevdokimov
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com <mailto:Viktor.Jevdokimov@adform.com>
> Phone: +370 5 212 3063 <tel:%2B370%205%20212%203063>
> Fax: +370 5 261 0453 <tel:%2B370%205%20261%200453>
>
> J. Jasinskio 16C,
> LT-01112 Vilnius,
> Lithuania
>
>
>
> Disclaimer: The information contained in this message and attachments 
> is intended solely for the attention and use of the named addressee 
> and may be confidential. If you are not the intended recipient, you 
> are reminded that the information remains the property of the sender. 
> You must not use, disclose, distribute, copy, print or rely on this 
> e-mail. If you have received this message in error, please contact the 
> sender immediately and irrevocably delete this message and any copies.
>


Mime
View raw message