cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Nickerson <pgn...@gmail.com>
Subject Re: Out of Memory Error While Opening SSTables on Startup
Date Tue, 10 Feb 2015 22:04:37 GMT
I was having trouble with snapshots failing while trying to repair that
table (http://www.mail-archive.com/user@cassandra.apache.org/msg40686.html).
I have a repair running on it now, and it seems to be going successfully
this time. I am going to wait for that to finish, then try a
manual nodetool compact. If that goes successfully, then would it be safe
to chalk the lack of compaction on this table in the past up to 2.1.2
problems?


 ~ Paul Nickerson

On Tue, Feb 10, 2015 at 3:34 PM, Chris Lohfink <clohfink85@gmail.com> wrote:

> Your cluster is probably having issues with compactions (with STCS you
> should never have this many).  I would probably punt with
> OpsCenter/rollups60. Turn the node off and move all of the sstables off to
> a different directory for backup (or just rm if you really don't care about
> 1 minute metrics), than turn the server back on.
>
> Once you get your cluster running again go back and investigate why
> compactions stopped, my guess is you hit an exception in past that killed
> your CompactionExecutor and things just built up slowly until you got to
> this point.
>
> Chris
>
> On Tue, Feb 10, 2015 at 2:15 PM, Paul Nickerson <pgn674@gmail.com> wrote:
>
>> Thank you Rob. I tried a 12 GiB heap size, and still crashed out. There
>> are 1,617,289 files under OpsCenter/rollups60.
>>
>> Once I downgraded Cassandra to 2.1.1 (apt-get install cassandra=2.1.1), I
>> was able to start up Cassandra OK with the default heap size formula.
>>
>> Now my cluster is running multiple versions of Cassandra. I think I will
>> downgrade the rest to 2.1.1.
>>
>>  ~ Paul Nickerson
>>
>> On Tue, Feb 10, 2015 at 2:05 PM, Robert Coli <rcoli@eventbrite.com>
>> wrote:
>>
>>> On Tue, Feb 10, 2015 at 11:02 AM, Paul Nickerson <pgn674@gmail.com>
>>> wrote:
>>>
>>>> I am getting an out of memory error why I try to start Cassandra on one
>>>> of my nodes. Cassandra will run for a minute, and then exit without
>>>> outputting any error in the log file. It is happening while SSTableReader
>>>> is opening a couple hundred thousand things.
>>>>
>>> ...
>>>
>>>> Does anyone know how I might get Cassandra on this node running again?
>>>> I'm not very familiar with correctly tuning Java memory parameters, and I'm
>>>> not sure if that's the right solution in this case anyway.
>>>>
>>>
>>> Try running 2.1.1, and/or increasing heap size beyond 8gb.
>>>
>>> Are there actually that many SSTables on disk?
>>>
>>> =Rob
>>>
>>>
>>
>>
>

Mime
View raw message