incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Row cache off-heap ?
Date Thu, 14 Mar 2013 13:35:53 GMT
> Should I raise a ticket since we are at least 3 having this issue from what I saw in the
mailing list ?
Sure, if you can come up with steps to reproduce the problem. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 14/03/2013, at 12:46 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:

> Thanks, I'll let you know when I'll do so. But any Idea about the increase of the heap
used if all seems to be well configured ? Should I raise a ticket since we are at least 3
having this issue from what I saw in the mailing list ?
> 
> 
> 2013/3/14 aaron morton <aaron@thelastpickle.com>
> > No, I didn't. I used the nodetool setcachecapacity and didn't restart the node.
> ok.
> 
> > I find them hudge, and just happened on the node in which I had enabled row cache.
I just enabled it on .164 node from 10:45 to 10:48 and the heap size doubled from 3.5GB to
7GB (out of 8, which induced memory pressure). About GC, all the collections increased a lot
compare to the other nodes with row caching disabled.
> 
> If the row cache provider is set to serialising, and the node restarted, under 1.1X it
will us the off heap cache.
> 
> At start up look for the log line
> "Initializing row cache with capacity of {} MBs and provider {"
> 
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 12/03/2013, at 1:44 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
> 
> > I am using C*1.1.6.
> >
> > "Did you restart the node after changing the row_cache_size_in_mb ?"
> >
> > No, I didn't. I used the nodetool setcachecapacity and didn't restart the node.
> >
> > "The changes in GC activity are not huge and may not be due to cache activity"
> >
> > I find them hudge, and just happened on the node in which I had enabled row cache.
I just enabled it on .164 node from 10:45 to 10:48 and the heap size doubled from 3.5GB to
7GB (out of 8, which induced memory pressure). About GC, all the collections increased a lot
compare to the other nodes with row caching disabled.
> >
> > "What is the output from nodetool info?"
> >
> > I can give it to you but, row cache i now disabled.
> >
> > Token            : 85070591730234615865843651857942052864
> > Gossip active    : true
> > Thrift active    : true
> > Load             : 201.61 GB
> > Generation No    : 1362749056
> > Uptime (seconds) : 328675
> > Heap Memory (MB) : 5157.58 / 8152.00
> > Data Center      : eu-west
> > Rack             : 1b
> > Exceptions       : 24
> > Key Cache        : size 104857584 (bytes), capacity 104857584 (bytes), 106814132
hits, 120131310 requests, 0.858 recent hit rate, 14400 save period in seconds
> > Row Cache        : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent
hit rate, 0 save period in seconds
> >
> > I think it won't help, but I can't try things now unless we are quire sure it will
work smooth, we are on heavy load.
> >
> > Anyway, thanks for trying to help once again.
> >
> >
> >
> >
> > 2013/3/12 aaron morton <aaron@thelastpickle.com>
> > What version are you using?
> >
> > Sounds like you have configured it correctly. Did you restart the node after changing
the row_cache_size_in_mb ?
> > The changes in GC activity are not huge and may not be due to cache activity. Have
they continued after you enabled the row cache?
> >
> > What is the output from nodetool info?
> >
> > Cheers
> >
> > -----------------
> > Aaron Morton
> > Freelance Cassandra Consultant
> > New Zealand
> >
> > @aaronmorton
> > http://www.thelastpickle.com
> >
> > On 11/03/2013, at 5:30 AM, Sávio Teles <savio.teles@lupa.inf.ufg.br> wrote:
> >
> >> I have the same problem!
> >>
> >> 2013/3/11 Alain RODRIGUEZ <arodrime@gmail.com>
> >> I can add that I have JNA corectly loaded, from the logs: "JNA mlockall successful"
> >>
> >>
> >> 2013/3/11 Alain RODRIGUEZ <arodrime@gmail.com>
> >> Any clue on this ?
> >>
> >> Row cache well configured could avoid us a lot of disk read, and IO is definitely
our bottleneck... If someone could explain why the row cache has so much impact on my JVM
and how to avoid it, it would be appreciated :).
> >>
> >>
> >> 2013/3/8 Alain RODRIGUEZ <arodrime@gmail.com>
> >> Hi,
> >>
> >> We have some issue having a high read throughput. I wanted to alleviate things
by turning the row cache ON.
> >>
> >> I set the row cache to 200 on one node and enable caching 'ALL' on the 3 most
read CF. There is the effect this operation had on my JVM: http://img692.imageshack.us/img692/4171/datastaxopscenterr.png
> >>
> >> It looks like the row cache was somehow stored in-heap. I looked at my cassandra.yaml
and I have the following configuration: row_cache_provider: SerializingCacheProvider (which
should be enough to store row cache off-heap as described above in this file: "SerializingCacheProvider
serialises the contents of the row and stores it in native memory, i.e., off the JVM Heap")
> >>
> >> What's wrong ?
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Atenciosamente,
> >> Sávio S. Teles de Oliveira
> >> voice: +55 62 9136 6996
> >> http://br.linkedin.com/in/savioteles
> >> Mestrando em Ciências da Computação - UFG
> >> Arquiteto de Software
> >> Laboratory for Ubiquitous and Pervasive Applications (LUPA) - UFG
> >
> >
> 
> 
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> 


Mime
View raw message