incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: OOM recovering failed node with many CFs
Date Thu, 26 May 2011 20:52:24 GMT
We've applied a fix to the 0.7 branch in
https://issues.apache.org/jira/browse/CASSANDRA-2714.  The patch
probably applies to 0.7.6 as well.

On Thu, May 26, 2011 at 11:36 AM, Flavio Baronti
<f.baronti@list-group.com> wrote:
> I tried the manual copy you suggest, but the SystemTable.checkHealth()
> function
> complains it can't load the system files. Log follows, I will gather some
> more
> info and create a ticket as soon as possible.
>
>  INFO [main] 2011-05-26 18:25:36,147 AbstractCassandraDaemon.java Logging
> initialized
>  INFO [main] 2011-05-26 18:25:36,172 AbstractCassandraDaemon.java Heap size:
> 4277534720/4277534720
>  INFO [main] 2011-05-26 18:25:36,174 CLibrary.java JNA not found. Native
> methods will be disabled.
>  INFO [main] 2011-05-26 18:25:36,190 DatabaseDescriptor.java Loading
> settings from file:/C:/Cassandra/conf/hscassandra9170.yaml
>  INFO [main] 2011-05-26 18:25:36,344 DatabaseDescriptor.java DiskAccessMode
> 'auto' determined to be mmap, indexAccessMode is mmap
>  INFO [main] 2011-05-26 18:25:36,532 SSTableReader.java Opening
> G:\Cassandra\data\system\Schema-f-2746
>  INFO [main] 2011-05-26 18:25:36,577 SSTableReader.java Opening
> G:\Cassandra\data\system\Schema-f-2729
>  INFO [main] 2011-05-26 18:25:36,590 SSTableReader.java Opening
> G:\Cassandra\data\system\Schema-f-2745
>  INFO [main] 2011-05-26 18:25:36,599 SSTableReader.java Opening
> G:\Cassandra\data\system\Migrations-f-2167
>  INFO [main] 2011-05-26 18:25:36,600 SSTableReader.java Opening
> G:\Cassandra\data\system\Migrations-f-2131
>  INFO [main] 2011-05-26 18:25:36,602 SSTableReader.java Opening
> G:\Cassandra\data\system\Migrations-f-1041
>  INFO [main] 2011-05-26 18:25:36,603 SSTableReader.java Opening
> G:\Cassandra\data\system\Migrations-f-1695
> ERROR [main] 2011-05-26 18:25:36,634 AbstractCassandraDaemon.java Fatal
> exception during initialization
> org.apache.cassandra.config.ConfigurationException: Found system table
> files, but they couldn't be loaded. Did you change the partitioner?
>        at
> org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:236)
>        at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:127)
>        at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314)
>        at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
>
>
> Il 5/26/2011 6:04 PM, Jonathan Ellis ha scritto:
>>
>> Sounds like a legitimate bug, although looking through the code I'm
>> not sure what would cause a tight retry loop on migration
>> announce/rectify. Can you create a ticket at
>> https://issues.apache.org/jira/browse/CASSANDRA ?
>>
>> As a workaround, I would try manually copying the Migrations and
>> Schema sstable files from the system keyspace of the live node, then
>> restart the recovering one.
>>
>> On Thu, May 26, 2011 at 9:27 AM, Flavio Baronti
>> <f.baronti@list-group.com>  wrote:
>>>
>>> I can't seem to be able to recover a failed node on a database where i
>>> did
>>> many updates to the schema.
>>>
>>> I have a small cluster with 2 nodes, around 1000 CF (I know it's a lot,
>>> but
>>> it can't be changed right now), and ReplicationFactor=2.
>>> I shut down a node and cleaned its data entirely, then tried to bring it
>>> back up. The node starts fetching schema updates from the live node, but
>>> the
>>> operation fails halfway with an OOME.
>>> After some investigation, what I found is that:
>>>
>>> - I have a lot of schema updates (there are 2067 rows in the
>>> system.Schema
>>> CF).
>>> - The live node loads migrations 1-1000, and sends them to the recovering
>>> node (Migration.getLocalMigrations())
>>> - Soon afterwards, the live node checks the schema version on the
>>> recovering
>>> node and finds it has moved by a little - say it has applied the first 3
>>> migrations. It then loads migrations 3-1003, and sends them to the node.
>>> - This process is repeated very quickly (sends migrations 6-1006, 9-1009,
>>> etc).
>>>
>>> Analyzing the memory dump and the logs, it looks like each of these 1000
>>> migration blocks are composed in a single message and sent to the
>>> OutboundTcpConnection queue. However, since the schema is big, the
>>> messages
>>> occupy a lot of space, and are built faster than the connection can send
>>> them. Therefore, they accumulate in OutboundTcpConnection.queue, until
>>> memory is completely filled.
>>>
>>> Any suggestions? Can I change something to make this work, apart from
>>> reducing the number of CFs?
>>>
>>> Flavio
>>>
>>
>>
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message