incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gurpreet Singh <gurpreet.si...@gmail.com>
Subject Re: cassandra read latency help
Date Fri, 18 May 2012 07:07:08 GMT
Hi Viktor,

As i mentioned, my goal is to finally achieve 100 reads per second
throughput, its not a batch size of 100.

The writes are already done, and i am not doing them anymore. I loaded the
system with about 130 million keys.

I am just doing a read workload in my experiment as of now..

1. no invalidation of cache happenning because no writes are happenning. GC
is not an issue, its an off-heap native cache by  default in 1.0.9
2. reads are with batch size 1
3. read qps 25

I am keeping max read qps constant, and just varying the number of threads
doing the reads.

row cache hit ratio = 0.66

Observations:

1. With 20 threads doing reads, avg latency is 50 ms
2. With 6 threads doing reads, avg latency is 30 ms
3. With 2 threads doing reads, avg latency is 15 ms
4. With 3 threads, latency is 20 ms

Looks like the number of disks (2) are limiting the concurrency of the
system here. any other explanations?
/G


On Thu, May 17, 2012 at 10:49 PM, Viktor Jevdokimov <
Viktor.Jevdokimov@adform.com> wrote:

>  Row cache is ok until keys are not heavily updated, otherwise it
> frequently invalidates and pressures GC.****
>
> ** **
>
> The high latency is from your batch of 100 keys. Review your data model to
> avoid such reads, if you need low latency.****
>
> ** **
>
> 500M rows on one node, or on the cluster? Reading 100 random rows at total
> of 40KB data from a data set of 180GB uncompressed under 30ms is not an
> easy task.****
>
> ** **
>
> ** **
>
>
>    Best regards / Pagarbiai
> *Viktor Jevdokimov*
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com
> Phone: +370 5 212 3063, Fax +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> Follow us on Twitter: @adforminsider <http://twitter.com/#!/adforminsider>
> What is Adform: watch this short video <http://vimeo.com/adform/display>
>  [image: Adform News] <http://www.adform.com>
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>   *From:* Gurpreet Singh [mailto:gurpreet.singh@gmail.com]
> *Sent:* Thursday, May 17, 2012 20:24
> *To:* user@cassandra.apache.org
> *Subject:* Re: cassandra read latency help****
>
> ** **
>
> Thanks Viktor for the advice.****
>
> Right now, i just have 1 node that i am testing against and i am using CL
> one.****
>
> Are you suggesting that the page cache might be doing better than the row
> cache?
> I am getting row cache hit of 0.66 right now.****
>
> ** **
>
> /G****
>
> ** **
>
> On Thu, May 17, 2012 at 12:26 AM, Viktor Jevdokimov <
> Viktor.Jevdokimov@adform.com> wrote:****
>
> > Gurpreet Singh wrote:
> > Any ideas on what could help here bring down the read latency even more ?
> ****
>
> Avoid Cassandra forwarding request to other nodes:
> - Use consistency level ONE;
> - Create data model to do single request with single key, since different
> keys may belong to different nodes and requires forwarding requests to them;
> - Use smart client to calculate token for key and select appropriate node
> (primary or replica) by token range;
> - Turn off Dynamic Snitch (it may forward request to other replica even it
> has the data);
> - Have all or hot data in page cache (no HDD disk IO) or use SSD;
> - If you do regular updates to key, do not use row cache, otherwise you
> may try.
>
>
>
>
> Best regards / Pagarbiai
>
> Viktor Jevdokimov
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com
> Phone: +370 5 212 3063
> Fax: +370 5 261 0453
>
> J. Jasinskio 16C,
> LT-01112 Vilnius,
> Lithuania
>
>
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.****
>
> ** **
>

Mime
View raw message