incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Hobbs <ty...@datastax.com>
Subject Re: OOM when applying migrations
Date Thu, 20 Sep 2012 23:06:11 GMT
This should explain the schema issue in 1.0 that has been fixed in 1.1:
http://www.datastax.com/dev/blog/the-schema-management-renaissance

On Thu, Sep 20, 2012 at 10:17 AM, Jason Wee <peichieh@gmail.com> wrote:

> Hi, when the heap is going more than 70% usage, you should be able to see
> in the log, many flushing, or reducing the row cache size down. Did you
> restart the cassandra daemon in the node that thrown OOM?
>
>
> On Thu, Sep 20, 2012 at 9:11 PM, Vanger <discipe@gmail.com> wrote:
>
>>  Hello,
>> We are trying to add new nodes to our *6-node* cassandra cluster with
>> RF=3 cassandra version 1.0.11. We are *adding 18 new nodes* one-by-one.
>>
>> First strange thing, I've noticed, is the number of completed
>> MigrationStage in nodetool tpstats grows for every new node, while schema
>> is not changed. For now with 21-nodes ring, for final join it shows 184683
>> migrations, while with 7-nodes it was about 50k migrations.
>> In fact it seems that this number is not a number of applied migrations.
>> When i grep log file with
>> grep "Applying migration" /var/log/cassandra/system.log -c
>> For each new node result is pretty much the same - around 7500 "Applying
>> migration" found in log.
>>
>> And the real problem is that now new nodes fail with Out Of Memory while
>> building schema from migrations. In logs we can find the following:
>>
>> WARN [ScheduledTasks:1] 2012-09-19 18:51:22,497 GCInspector.java (line
>> 145) Heap is 0.7712290960125684 full.  You may need to reduce memtable
>> and/or cache sizes.  Cassandra will now flush up to the two largest
>> memtables to free up memory.  Adjust flush_largest_memtables_at threshold
>> in cassandra.yaml if you don't want Cassandra to do this automatically
>>  INFO [ScheduledTasks:1] 2012-09-19 18:51:22,498 StorageService.java
>> (line 2658) Unable to reduce heap usage since there are no dirty column
>> families
>> ....
>>  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 GCInspector.java (line
>> 139) Heap is 0.853078131310858 full.  You may need to reduce memtable
>> and/or cache sizes.  Cassandra is now reducing cache sizes to free up
>> memory.  Adjust reduce_cache_sizes_at threshold in cassandra.yaml if you
>> don't want Cassandra to do this automatically
>>  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 AutoSavingCache.java
>> (line 187) Reducing AppUser RowCache capacity from 100000 to 0 to reduce
>> memory pressure
>>  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 AutoSavingCache.java
>> (line 187) Reducing AppUser KeyCache capacity from 100000 to 0 to reduce
>> memory pressure
>>  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 AutoSavingCache.java
>> (line 187) Reducing PaymentClaim KeyCache capacity from 50000 to 0 to
>> reduce memory pressure
>>  WARN [ScheduledTasks:1] 2012-09-19 18:51:29,500 AutoSavingCache.java
>> (line 187) Reducing Organization RowCache capacity from 1000 to 0 to reduce
>> memory pressure
>>  .....
>>  INFO [main] 2012-09-19 18:57:14,181 StorageService.java (line 668)
>> JOINING: waiting for schema information to complete
>> ERROR [Thread-28] 2012-09-19 18:57:14,198 AbstractCassandraDaemon.java
>> (line 139) Fatal exception in thread Thread[Thread-28,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>>         at
>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:140)
>>         at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:115)
>> ...
>> ERROR [ReadStage:353] 2012-09-19 18:57:20,453
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[ReadStage:353,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>>         at
>> org.apache.cassandra.service.MigrationManager.makeColumns(MigrationManager.java:256)
>>         at
>> org.apache.cassandra.db.DefinitionsUpdateVerbHandler.doVerb(DefinitionsUpdateVerbHandler.java:51)
>>         at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>
>>
>> Originally "max heap size" was set to 6G. Then we increased heap size
>> limit to 8G and it works. But warnings still present
>>
>>  WARN [ScheduledTasks:1] 2012-09-20 11:39:11,373 GCInspector.java (line
>> 145) Heap is 0.7760745735786222 full.  You may need to reduce memtable
>> and/or cache sizes.  Cassandra will now flush up to the two largest
>> memtables to free up memory.  Adjust flush_largest_memtables_at threshold
>> in cassandra.yaml if you don't want Cassandra to do this automatically
>>  INFO [ScheduledTasks:1] 2012-09-20 11:39:11,374 StorageService.java
>> (line 2658) Unable to reduce heap usage since there are no dirty column
>> families
>>
>> It is probably a bug in applying migrations.
>> Could anyone explain why cassandra behaves this way? Could you please
>> recommend us smth to cope with this situation?
>> Thank you in advance.
>>
>> --
>> W/ best regards,
>> Sergey B.
>>
>>
>


-- 
Tyler Hobbs
DataStax <http://datastax.com/>

Mime
View raw message