cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Spriegel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7401) Memtable.maybeUpdateLiveRatio goes into an endless loop when currentOperations is zero
Date Mon, 14 Jul 2014 07:35:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14060404#comment-14060404
] 

Christian Spriegel commented on CASSANDRA-7401:
-----------------------------------------------

I just found my log message in our logfiles (where my custom-build is running):

{quote}
WARN [MutationStage:43] 2014-07-13 18:06:11,365 Memtable.java (line 184) TEST - This would
be a crash: Memtable-ResponsePortal@908155793(0/0 serialized/live bytes, 0 ops)

 WARN [MutationStage:46] 2014-07-13 18:06:11,365 Memtable.java (line 184) TEST - This would
be a crash: Memtable-ResponsePortal@908155793(0/0 serialized/live bytes, 0 ops)
{quote}

It shows that the Memtable can be empty. Why that is, I have no idea.

Nevertheless, the fix works.

> Memtable.maybeUpdateLiveRatio goes into an endless loop when currentOperations is zero
> --------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7401
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7401
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Christian Spriegel
>            Assignee: Christian Spriegel
>             Fix For: 2.0.9
>
>         Attachments: MemtableFixV1.patch
>
>
> Hi,
> I was describing an error the other day on the mailing list, where the MemoryMeter would
go into an endless loop. This happened multiple times last week, unfortunetaly I cannot reproduce
it at the moment.
> The whole cassandra server got unresponsive and logged about 7000k messages per second
into the log:
> {quote}
> ...
>  INFO [MemoryMeter:1] 2014-06-14 19:24:09,488 Memtable.java (line 481) CFS(Keyspace='MDS',
ColumnFamily='ResponsePortal') liveRatio is 64.0 (just-counted was 64.0).  calculation took
0ms for 0 cells
> ...
> {quote}
> The cause for this seems to be Memtable.maybeUpdateLiveRatio(), which cannot handle currentOperations
(and liveRatioComputedAt) to be zero. The loop will iterate endlessly:
> {code}
>             ...
>             if (operations < 2 * last) // does never break when zero: 0 < 0 is
not true
>                 break;
>             ...
> {code}
> One thing I cannot explain: How can the operationcount be zero when maybeUpdateLiveRatio()
gets called?
> is it possible that addAndGet in resolve() increases by 0  in some cases?
> {code}
> currentOperations.addAndGet(cf.getColumnCount() + (cf.isMarkedForDelete() ? 1 : 0) +
cf.deletionInfo().rangeCount()); // can this be zero? 
> {code}
> Nevertheless, the attached patch fixes the endless loop. Feel free to reassign this ticket
or create a followup ticket if currentOperations should not be zero.
> kind regards,
> Christian



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message