cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charlie Moad <>
Subject Re: OutOfMemoryError when initializing a secondary index
Date Tue, 09 Aug 2016 15:01:31 GMT
I was able to get a heap dump from one of the machines and discovered that
where the console was stopping every time was not actually the cause of the
problem. It was another secondary index that was listed earlier in the
console. I was able to start the nodes successfully once I moved that table
and corresponding indexes out of the way.

It is still concerning that there was an effective memory leak or infinite
loop when trying to restore the secondary index. You can see from the
screenshot that the "oac.db.CoumnFamilyStore" object stacked up to 1.8GB,
filling the heap.

[image: Inline image 1]

On Tue, Aug 9, 2016 at 8:33 AM, Carlos Alonso <> wrote:

> If you're deleting all traces of the index you probably want to look at
> the commit log as they are probably being recreated from there.
> Hope it helps.
> Carlos Alonso | Software Engineer | @calonso <>
> On 5 August 2016 at 23:05, Charlie Moad <>
> wrote:
>> Running Cassandra 3.0.7 we have 3 out of 6 nodes that threw an OOM error
>> when a developer created a secondary index. I'm trying to repair the
>> cluster. I stopped all nodes, deleted all traces of the table and secondary
>> index from disk, removed commit logs and saved caches, and restarted the
>> instances. The 3 nodes that didn't have the OOM error started fine, but the
>> other three are getting stuck while trying to initialize the secondary
>> index – which shouldn't even have data to load.
>> """
>> ...
>> INFO  19:51:59 Initializing notifications_v1.notifications_tray
>> INFO  19:51:59 Initializing notifications_v1.notifications
>> _tray.notifications_tray_event_id
>> """
>> The instances spin for a long time then throw an OutOfMemoryError.
>> I don't need to save this table, but I do need to save other keyspaces.
>> Is there any way I can get these nodes operational again?

*Charlie Moad* | Director of Production Operations

[image:] <> (o) 317.661.4897
(c) 317.366.8687 <>

55 Monument Circle, Suite 600, Indianapolis, IN 46204

[image: LinkedIn] <>


[image: Powered by Sigstr]

View raw message