Return-Path: 
 <commits-return-103848-apmail-cassandra-commits-archive=cassandra.apache.org@cassandra.apache.org>
X-Original-To: apmail-cassandra-commits-archive@www.apache.org
Delivered-To: apmail-cassandra-commits-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB77A11A59
	for <apmail-cassandra-commits-archive@www.apache.org>;
 Sat, 28 Jun 2014 17:32:24 +0000 (UTC)
Received: (qmail 42092 invoked by uid 500); 28 Jun 2014 17:32:24 -0000
Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org
Received: (qmail 42051 invoked by uid 500); 28 Jun 2014 17:32:24 -0000
Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:commits-help@cassandra.apache.org>
List-Unsubscribe: <mailto:commits-unsubscribe@cassandra.apache.org>
List-Post: <mailto:commits@cassandra.apache.org>
List-Id: <commits.cassandra.apache.org>
Reply-To: dev@cassandra.apache.org
Delivered-To: mailing list commits@cassandra.apache.org
Received: (qmail 42038 invoked by uid 99); 28 Jun 2014 17:32:24 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Jun 2014 17:32:24 +0000
Date: Sat, 28 Jun 2014 17:32:24 +0000 (UTC)
From: "Jackson Chung (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12724298.1403976629101.68540.1403976744490@arcas>
In-Reply-To: <JIRA.12724298.1403976629101@arcas>
References: <JIRA.12724298.1403976629101@arcas>
Subject: [jira] [Created] (CASSANDRA-7467) flood of "setting live ratio to
 maximum of 64" from repair
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Jackson Chung created CASSANDRA-7467:
----------------------------------------

             Summary: flood of "setting live ratio to maximum of 64" from repair
                 Key: CASSANDRA-7467
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7467
             Project: Cassandra
          Issue Type: Bug
            Reporter: Jackson Chung


we are on 2.0.8

running with repair -pr -local <KS>, all nodes on i2.2x (60G ram);, with setting 8G of heap. Using java 8. (key cache size is 1G)

On occasion, when repair is run, the C* that run the repair, or another node in the cluster, or both, run into a bad state with the system.log just printing ""setting live ratio to maximum of 64"  forever every split seconds. It usually happens when repairing one of the larger/wider CF. 

 WARN [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 470) setting live ratio to maximum of 64.0 instead of Infinity
 INFO [MemoryMeter:1] 2014-06-28 09:13:24,540 Memtable.java (line 481) CFS(Keyspace='RIQ', ColumnFamily='MemberTimeline') liveRatio is 64.0 (just-counted was 64.0).  calculation took 0ms for 0 cells

		Table: MemberTimeline
		SSTable count: 13
		Space used (live), bytes: 17644018786
...
		Compacted partition minimum bytes: 30
		Compacted partition maximum bytes: 464228842
		Compacted partition mean bytes: 54578

Just to give an idea of how bad this is, the log file is set to rotate 50 times with 21M each. In less than 15 minutes, all the logs are filled up with just that log. C* is not responding, and can't be killed normally. Only way is to kill -9



--
This message was sent by Atlassian JIRA
(v6.2#6252)