incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nate McCall <n...@thelastpickle.com>
Subject Re: Cassandra shuts down; was:Cassandra crashes
Date Wed, 04 Sep 2013 15:11:16 GMT
Ideally, you should get back pressure in the form of dropped messages
before you see crashes, but if turning down the heap allocation was the
only thing you did, there are other changes required (several mentioned by
Romain above are very good places to start).
A few other ideas:
- did you adjust ParNew along with heap?
- you may want to adjust SurvivorRatio and MaxTenuringThreshold (change
both to 4 as a starting point) in JVM
- definitely play with compaction throughput by turning it way up since you
have IO capacity

These will cause you to GC and compact continuously in this environment,
but it should at least keep going


On Wed, Sep 4, 2013 at 9:14 AM, Romain HARDOUIN
<romain.hardouin@urssaf.fr>wrote:

> Have you tried to tweak settings like memtable_total_space_in_mb and
> flush_largest_memtables_at?
> Also, the compaction manager seems to be pretty busy, take a look at
> in_memory_compaction_limit_in_mb.
> And with SSD hardware you should modifiy multithreaded_compaction,
> compaction_throughput_mb_per_sec, concurrent_reads and concurrent_writes.
> Of course 2GB of RAM is low but tweak these settings might help you.
> Maybe some guru could confirm/infirm that.
>
>
>
>
> De :        Jan Algermissen <jan.algermissen@nordsc.com>
> A :        user@cassandra.apache.org,
> Date :        04/09/2013 12:29
> Objet :        Re: Cassandra shuts down; was:Cassandra crashes
> ------------------------------
>
>
>
> Romain,
>
>
> On 04.09.2013, at 11:11, Romain HARDOUIN <romain.hardouin@urssaf.fr>
> wrote:
>
> > Maybe you should include the end of Cassandra logs.
>
> There is nothing that seems interesting in cassandra.log. Below you find
> system.log.
>
> > What comes to my mind when I read your first post is OOM killer.
> > But what you describe later is not the case.
> > Just to be sure, have you checked /var/log/messages?
>
> Nothing there, just occasional Firewall TCP rejections.
>
> Somehow I think I am simply overloading the whole cluster (see the hinted
> handoff messages in the log). Could that be due to the limited memory of
> 2GB my nodes have? IOW, not enough space to buffer up the writes before
> dumping to disk?
>
> Also, my overall write performance is actually pretty bad compared to what
> I read about C*. Before I thought it was the client doing to much work or
> the network. Turns out that's not the case.
>
> I'd expect C* to sort of just suck in my rather small amount of data -
> must be me, not using the right configuration. Oh well, I'll get there :-)
> Thanks anyhow.
>
> Jan
>
>
>
>
>

Mime
View raw message