incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vanger <disc...@gmail.com>
Subject OOM when applying migrations
Date Thu, 20 Sep 2012 13:11:16 GMT
Hello,
We are trying to add new nodes to our *6-node* cassandra cluster with 
RF=3 cassandra version 1.0.11. We are *adding 18 new nodes* one-by-one.

First strange thing, I've noticed, is the number of completed 
MigrationStage in nodetool tpstats grows for every new node, while 
schema is not changed. For now with 21-nodes ring, for final join it 
shows 184683 migrations, while with 7-nodes it was about 50k migrations.
In fact it seems that this number is not a number of applied migrations. 
When i grep log file with
grep "Applying migration" /var/log/cassandra/system.log -c
For each new node result is pretty much the same - around 7500 "Applying 
migration" found in log.

And the real problem is that now new nodes fail with Out Of Memory while 
building schema from migrations. In logs we can find the following:

WARN [ScheduledTasks:1] 2012-09-19 18:51:22,497 GCInspector.java (line 
145) Heap is 0.7712290960125684 full.  You may need to reduce memtable 
and/or cache sizes. Cassandra will now flush up to the two largest 
memtables to free up memory.  Adjust flush_largest_memtables_at 
threshold in cassandra.yaml if you don't want Cassandra to do this 
automatically
  INFO [ScheduledTasks:1] 2012-09-19 18:51:22,498 StorageService.java 
(line 2658) Unable to reduce heap usage since there are no dirty column 
families
....
  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 GCInspector.java (line 
139) Heap is 0.853078131310858 full. You may need to reduce memtable 
and/or cache sizes. Cassandra is now reducing cache sizes to free up 
memory. Adjust reduce_cache_sizes_at threshold in cassandra.yaml if you 
don't want Cassandra to do this automatically
  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 AutoSavingCache.java 
(line 187) Reducing AppUser RowCache capacity from 100000 to 0 to reduce 
memory pressure
  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 AutoSavingCache.java 
(line 187) Reducing AppUser KeyCache capacity from 100000 to 0 to reduce 
memory pressure
  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 AutoSavingCache.java 
(line 187) Reducing PaymentClaim KeyCache capacity from 50000 to 0 to 
reduce memory pressure
  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 AutoSavingCache.java 
(line 187) Reducing Organization RowCache capacity from 1000 to 0 to 
reduce memory pressure
  .....
  INFO [main] 2012-09-19 18:57:14,181 StorageService.java (line 668) 
JOINING: waiting for schema information to complete
ERROR [Thread-28] 2012-09-19 18:57:14,198 AbstractCassandraDaemon.java 
(line 139) Fatal exception in thread Thread[Thread-28,5,main]
java.lang.OutOfMemoryError: Java heap space
         at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:140)
         at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:115)
...
ERROR [ReadStage:353] 2012-09-19 18:57:20,453 
AbstractCassandraDaemon.java (line 139) Fatal exception in thread 
Thread[ReadStage:353,5,main]
java.lang.OutOfMemoryError: Java heap space
         at 
org.apache.cassandra.service.MigrationManager.makeColumns(MigrationManager.java:256)
         at 
org.apache.cassandra.db.DefinitionsUpdateVerbHandler.doVerb(DefinitionsUpdateVerbHandler.java:51)
         at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)


Originally "max heap size" was set to 6G. Then we increased heap size 
limit to 8G and it works. But warnings still present

  WARN [ScheduledTasks:1] 2012-09-20 11:39:11,373 GCInspector.java (line 
145) Heap is 0.7760745735786222 full. You may need to reduce memtable 
and/or cache sizes.  Cassandra will now flush up to the two largest 
memtables to free up memory.  Adjust flush_largest_memtables_at 
threshold in cassandra.yaml if you don't want Cassandra to do this 
automatically
  INFO [ScheduledTasks:1] 2012-09-20 11:39:11,374 StorageService.java 
(line 2658) Unable to reduce heap usage since there are no dirty column 
families

It is probably a bug in applying migrations.
Could anyone explain why cassandra behaves this way? Could you please 
recommend us smth to cope with this situation?
Thank you in advance.

--
W/ best regards,
Sergey B.


Mime
View raw message