From user-return-21988-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu Nov 3 18:44:29 2011 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8D4C77669 for ; Thu, 3 Nov 2011 18:44:29 +0000 (UTC) Received: (qmail 27503 invoked by uid 500); 3 Nov 2011 18:44:27 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 27476 invoked by uid 500); 3 Nov 2011 18:44:27 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 27467 invoked by uid 99); 3 Nov 2011 18:44:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Nov 2011 18:44:27 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 209.85.215.172 as permitted sender) Received: from [209.85.215.172] (HELO mail-ey0-f172.google.com) (209.85.215.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Nov 2011 18:44:21 +0000 Received: by eyg24 with SMTP id 24so1662868eyg.31 for ; Thu, 03 Nov 2011 11:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=2FCeNQdsBMv8USuVNBhO6DPH3BXiOfQGbWFRSojibv8=; b=FHRWNVY8Xb/1fD1udA/a23R0ZQE5SIz7L0379RYnmBCgpm1VEHKnu5tTJPA0qBv9SP r4Ee2NQx400vESKuUKWScRg8qJ3jzI5aC5iL8jBjb6tAgLTgW9rS3U3LdmnlSPfGai+T cQZuHDkTo2f7aIkNzO7wcf1AAb+PdhLRsuN0Q= Received: by 10.213.28.68 with SMTP id l4mr1199045ebc.58.1320345841133; Thu, 03 Nov 2011 11:44:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.102.19 with HTTP; Thu, 3 Nov 2011 11:43:40 -0700 (PDT) In-Reply-To: <376CEC01195C894CB9F8A3C274029A96B13E83B0@FISH-EX2K10-01.azaleos.net> References: <376CEC01195C894CB9F8A3C274029A96B13E83B0@FISH-EX2K10-01.azaleos.net> From: Jonathan Ellis Date: Thu, 3 Nov 2011 13:43:40 -0500 Message-ID: Subject: Re: Problem after upgrade to 1.0.1 To: user@cassandra.apache.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Just to rule it out: you didn't do anything tricky like update HintsColumnFamily to use compression? On Thu, Nov 3, 2011 at 1:39 PM, Bryce Godfrey w= rote: > I recently upgraded from 0.8.6 to 1.0.1 and everything seemed to go just > fine with the rolling upgrade.=A0 But now I=92m having extreme load growt= h on > one of my nodes (and others are growing faster than usual also).=A0 I > attempted to run a cfstats against the extremely large node that was seei= ng > 2x the load of others and I get this error below.=A0 I=92m also went into= the > o.a.c.db.HintedHandoffManager mbean and attempted to list pending hints t= o > see if it was growing out of control for some reason, but that just times > out eventually for any node.=A0 I=92m not sure what to do next with this = issue. > > > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Column Family: HintsColumnFami= ly > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 SSTable count: 3 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Space used (live): 12681676= 437 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Space used (total): 1023313= 0272 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Number of Keys (estimate): = 384 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Memtable Columns Count: 117= 704 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Memtable Data Size: 1151073= 07 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Memtable Switch Count: 66 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Read Count: 0 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Read Latency: NaN ms. > > =A0=A0=A0=A0=A0 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0Write Count: 21203290 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Write Latency: 0.014 ms. > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Pending Tasks: 0 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Key cache capacity: 3 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Key cache size: 0 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Key cache hit rate: NaN > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Row cache: disabled > > =A0 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0Compacted row minimum size:= 30130993 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Compacted row maximum size:= 9223372036854775807 > > Exception in thread "main" java.lang.IllegalStateException: Unable to > compute ceiling for max when histogram overflowed > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.jav= a:170) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:395) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyStor= e.java:293) > > =A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Nat= ive Method) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :39) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:25) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanI= ntrospector.java:93) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanI= ntrospector.java:27) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:= 208) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Defaul= tMBeanServerInterceptor.java:666) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:6= 38) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionIm= pl.java:1404) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImp= l.java:72) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMI= ConnectionImpl.java:1265) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMICo= nnectionImpl.java:1360) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionI= mpl.java:600) > > =A0=A0=A0=A0=A0=A0=A0 at sun.reflect.GeneratedMethodAccessor15.invoke(Unk= nown Source) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:25) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > > =A0=A0=A0=A0=A0=A0=A0 at sun.rmi.transport.Transport$1.run(Transport.java= :159) > > =A0=A0=A0=A0=A0=A0=A0 at java.security.AccessController.doPrivileged(Nati= ve Method) > > =A0=A0=A0=A0=A0=A0=A0 at sun.rmi.transport.Transport.serviceCall(Transpor= t.java:155) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.ja= va:790) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.jav= a:649) > > =A0=A0=A0=A0=A0=A0=A0 at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886) > > =A0=A0=A0=A0=A0=A0=A0 at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:662) > > > > Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation=A0| T: > 206.926.1978 | M: 206.849.2477 > > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com