cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From horschi <hors...@gmail.com>
Subject Re: Cassandra 2.0.8 MemoryMeter goes crazy
Date Mon, 16 Jun 2014 10:04:43 GMT
Hi again,

before people start replying here: I just reported a Jira ticket:
https://issues.apache.org/jira/browse/CASSANDRA-7401

I think Memtable.maybeUpdateLiveRatio() needs some love.

kind regards,
Christian



On Sat, Jun 14, 2014 at 10:02 PM, horschi <horschi@gmail.com> wrote:

> Hi everyone,
>
> this week we upgraded one of our Systems from Cassandra 1.2.16 to 2.0.8.
> All 3 nodes were upgraded. SStables are upgraded.
>
> Unfortunetaly we are now experiencing that Cassandra starts to hang every
> 10 hours or so.
>
> We can see the MemoryMeter being very active, every time it is hanging.
> Both in tpstats and in the system.log:
>
>  INFO [MemoryMeter:1] 2014-06-14 19:24:09,488 Memtable.java (line 481)
> CFS(Keyspace='MDS', ColumnFamily='ResponsePortal') liveRatio is 64.0
> (just-counted was 64.0).  calculation took 0ms for 0 cells
>
> This line is logged hundreds of times per second (!) when Cassandra is
> down. CPU is a 100% busy.
>
> Interestingly this is only logged for this particular Columnfamily. This
> CF is used as a queue, which only contains a few entries (datafiles are
> about 4kb, only ~100 keys, usually 1-2 active, 98-99 tombstones).
>
>             Table: ResponsePortal
>             SSTable count: 1
>             Space used (live), bytes: 4863
>             Space used (total), bytes: 4863
>             SSTable Compression Ratio: 0.9545454545454546
>             Number of keys (estimate): 128
>             Memtable cell count: 0
>             Memtable data size, bytes: 0
>             Memtable switch count: 1
>             Local read count: 0
>             Local read latency: 0.000 ms
>             Local write count: 5
>             Local write latency: 0.000 ms
>             Pending tasks: 0
>             Bloom filter false positives: 0
>             Bloom filter false ratio: 0.00000
>             Bloom filter space used, bytes: 176
>             Compacted partition minimum bytes: 43
>             Compacted partition maximum bytes: 50
>             Compacted partition mean bytes: 50
>             Average live cells per slice (last five minutes): 0.0
>             Average tombstones per slice (last five minutes): 0.0
>
>
>         Table: ResponsePortal
>         SSTable count: 1
>         Space used (live), bytes: 4765
>         Space used (total), bytes: 5777
>         SSTable Compression Ratio: 0.75
>         Number of keys (estimate): 128
>         Memtable cell count: 0
>         Memtable data size, bytes: 0
>         Memtable switch count: 12
>         Local read count: 0
>         Local read latency: 0.000 ms
>         Local write count: 1096
>         Local write latency: 0.000 ms
>         Pending tasks: 0
>         Bloom filter false positives: 0
>         Bloom filter false ratio: 0.00000
>         Bloom filter space used, bytes: 16
>         Compacted partition minimum bytes: 43
>         Compacted partition maximum bytes: 50
>         Compacted partition mean bytes: 50
>         Average live cells per slice (last five minutes): 0.0
>         Average tombstones per slice (last five minutes): 0.0
>
>
> Has anyone ever seen this or has an idea what could be wrong? It seems
> that 2.0 can handle this column family not as good as 1.2 could.
>
> Any hints on what could be wrong are greatly appreciated :-)
>
> Cheers,
> Christian
>

Mime
View raw message