cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Baronti <f.baro...@list-group.com>
Subject OOM recovering failed node with many CFs
Date Thu, 26 May 2011 14:27:52 GMT
I can't seem to be able to recover a failed node on a database where i 
did many updates to the schema.

I have a small cluster with 2 nodes, around 1000 CF (I know it's a lot, 
but it can't be changed right now), and ReplicationFactor=2.
I shut down a node and cleaned its data entirely, then tried to bring it 
back up. The node starts fetching schema updates from the live node, but 
the operation fails halfway with an OOME.
After some investigation, what I found is that:

- I have a lot of schema updates (there are 2067 rows in the 
system.Schema CF).
- The live node loads migrations 1-1000, and sends them to the 
recovering node (Migration.getLocalMigrations())
- Soon afterwards, the live node checks the schema version on the 
recovering node and finds it has moved by a little - say it has applied 
the first 3 migrations. It then loads migrations 3-1003, and sends them 
to the node.
- This process is repeated very quickly (sends migrations 6-1006, 
9-1009, etc).

Analyzing the memory dump and the logs, it looks like each of these 1000 
migration blocks are composed in a single message and sent to the 
OutboundTcpConnection queue. However, since the schema is big, the 
messages occupy a lot of space, and are built faster than the connection 
can send them. Therefore, they accumulate in 
OutboundTcpConnection.queue, until memory is completely filled.

Any suggestions? Can I change something to make this work, apart from 
reducing the number of CFs?

Flavio

Mime
View raw message