From user-return-24739-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Mar 14 19:34:15 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4B7919082 for ; Wed, 14 Mar 2012 19:34:15 +0000 (UTC) Received: (qmail 56670 invoked by uid 500); 14 Mar 2012 19:34:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 56648 invoked by uid 500); 14 Mar 2012 19:34:12 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 56640 invoked by uid 99); 14 Mar 2012 19:34:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Mar 2012 19:34:12 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [216.82.241.99] (HELO mail51.messagelabs.com) (216.82.241.99) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Mar 2012 19:34:04 +0000 X-Env-Sender: Bryce.Godfrey@azaleos.com X-Msg-Ref: server-2.tower-51.messagelabs.com!1331753621!3940159!2 X-Originating-IP: [96.31.162.253] X-StarScan-Version: 6.5.5; banners=-,-,- X-VirusChecked: Checked Received: (qmail 1290 invoked from network); 14 Mar 2012 19:33:42 -0000 Received: from unknown (HELO smtp.azaleos.net) (96.31.162.253) by server-2.tower-51.messagelabs.com with AES128-SHA encrypted SMTP; 14 Mar 2012 19:33:42 -0000 Received: from FISH-EX2K10-01.azaleos.net ([fe80::389b:5e08:38cd:b989]) by FISH-EX2K10-02.azaleos.net ([fe80::92b:9330:6521:ef18%13]) with mapi id 14.02.0283.003; Wed, 14 Mar 2012 12:33:42 -0700 From: Bryce Godfrey To: "user@cassandra.apache.org" Subject: Large hints column family Thread-Topic: Large hints column family Thread-Index: Ac0CGOOSsB7ZiYpYTpqIVJjUcubtJw== Date: Wed, 14 Mar 2012 19:33:42 +0000 Message-ID: <376CEC01195C894CB9F8A3C274029A96B471C1F7@FISH-EX2K10-01.azaleos.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.200.10.162] Content-Type: multipart/alternative; boundary="_000_376CEC01195C894CB9F8A3C274029A96B471C1F7FISHEX2K1001aza_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_376CEC01195C894CB9F8A3C274029A96B471C1F7FISHEX2K1001aza_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The system HintsColumnFamily seems large in my cluster, and I want to track= down why that is. I try invoking "listEndpointsPendingHints()" for o.a.c.= db.HintedHandoffManager and it never returns, and also freezes the node tha= t its invoked against. It's a 3 node cluster, and all nodes have been up a= nd running without issue for a while. Any help on where to start with this= ? Column Family: HintsColumnFamily SSTable count: 11 Space used (live): 11271669539 Space used (total): 11271669539 Number of Keys (estimate): 1408 Memtable Columns Count: 338 Memtable Data Size: 0 Memtable Switch Count: 1 Read Count: 3 Read Latency: 4354.669 ms. Write Count: 848 Write Latency: 0.029 ms. Pending Tasks: 0 Bloom Filter False Postives: 0 Bloom Filter False Ratio: 0.00000 Bloom Filter Space Used: 12656 Key cache capacity: 14 Key cache size: 11 Key cache hit rate: 0.6666666666666666 Row cache: disabled Compacted row minimum size: 105779 Compacted row maximum size: 7152383774 Compacted row mean size: 590818614 Thanks, Bryce --_000_376CEC01195C894CB9F8A3C274029A96B471C1F7FISHEX2K1001aza_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

The system HintsColumnFamily seems large in my clust= er, and I want to track down why that is.  I try invoking “listE= ndpointsPendingHints()” for o.a.c.db.HintedHandoffManager and it neve= r returns, and also freezes the node that its invoked against.  It’s a 3 node cluster, and all nodes have been up and= running without issue for a while.  Any help on where to start with t= his?

 

        &nbs= p;      Column Family: HintsColumnFamily

        &nbs= p;       SSTable count: 11

        &nbs= p;       Space used (live): 11271669539<= /o:p>

        &nbs= p;       Space used (total): 11271669539=

        &nbs= p;       Number of Keys (estimate): 1408=

        &nbs= p;       Memtable Columns Count: 338

        &nbs= p;       Memtable Data Size: 0

        &nbs= p;       Memtable Switch Count: 1<= /p>

        &nbs= p;       Read Count: 3

        &nbs= p;       Read Latency: 4354.669 ms.

        &nbs= p;       Write Count: 848

        &nbs= p;       Write Latency: 0.029 ms.<= /p>

        &nbs= p;       Pending Tasks: 0

        &nbs= p;       Bloom Filter False Postives: 0<= /o:p>

        &nbs= p;       Bloom Filter False Ratio: 0.00000

        &nbs= p;       Bloom Filter Space Used: 12656<= /o:p>

        &nbs= p;       Key cache capacity: 14

        &nbs= p;       Key cache size: 11

        &nbs= p;       Key cache hit rate: 0.66666666666666= 66

        &nbs= p;       Row cache: disabled

        &nbs= p;       Compacted row minimum size: 105779

        &nbs= p;       Compacted row maximum size: 71523837= 74

        &nbs= p;       Compacted row mean size: 590818614

 

Thanks,

Bryce

--_000_376CEC01195C894CB9F8A3C274029A96B471C1F7FISHEX2K1001aza_--