incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Large system.Migration CF after upgrade to 1.1
Date Wed, 27 Nov 2013 02:03:05 GMT
> We have noticed that a cluster we upgraded to 1.1.6 (from 1.0.*) still has a single large
(~4GB) row in system.Migrations on each cluster node. 
There is some code in there to drop that CF at startup, but I’m not sure on the requirements
for it to run. if the time stamps have not been updated in a while copy them out of the way
and restart. 

> We are also seeing heap pressure / Full GC issues when we do schema updates to this cluster
How much memory does the machine have and how is the JVM configured ? 

On pre 1.1 that is often a result of memory pressure from the bloom filters and compression
meta data being on the JVM heap. Do you have a lot (i.e. > 500Million ) rows per node ?


Check how small CMS can get the heap, it may be the case that it just cannot reduce it further.


As a work around you can: increase the heap, increase bloom_filter_fp_chance (per cf) and
index_interval (yaml). My talk called “In case of emergency break glass” at the summit
in SF this year talks about this http://thelastpickle.com/speaking/2013/06/11/Speaking-Cassandra-Summit-SF-2013.html

Long term moving to 1.2 will help. 

Hope that helps. 

 
-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 23/11/2013, at 10:35 am, Andrew Cooper <Andrew.Cooper@nisc.coop> wrote:

> We have noticed that a cluster we upgraded to 1.1.6 (from 1.0.*) still has a single large
(~4GB) row in system.Migrations on each cluster node.  We are also seeing heap pressure /
Full GC issues when we do schema updates to this cluster.  If the two are related, is it possible
to somehow remove/truncate the system.Migrations CF?  If I understand correctly, version 1.1
no longer uses this CF, instead using the system.schema_* CF's.   We have multiple clusters
and clusters which were built from scratch at version 1.1 or 1.2 do no have data in system.Migrations.
> 
> I would appreciate any advice and I can provide more details if needed.  
> 
> -Andrew
> 
> Andrew Cooper
> National Information Solutions Cooperative®
> 3201 Nygren Drive NW
> Mandan, ND 58554
> + e-mail: andrew.cooper@nisc.coop
> ( phone: 866.999.6472 ext 6824
> ( direct: 701-667-6824  
> 


Mime
View raw message