Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9A3CA7813 for ; Thu, 3 Nov 2011 18:53:01 +0000 (UTC) Received: (qmail 44527 invoked by uid 500); 3 Nov 2011 18:52:59 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 44499 invoked by uid 500); 3 Nov 2011 18:52:59 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 44490 invoked by uid 99); 3 Nov 2011 18:52:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Nov 2011 18:52:59 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_FROM,FREEMAIL_REPLY,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 209.85.215.172 as permitted sender) Received: from [209.85.215.172] (HELO mail-ey0-f172.google.com) (209.85.215.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Nov 2011 18:52:53 +0000 Received: by eyg24 with SMTP id 24so1672718eyg.31 for ; Thu, 03 Nov 2011 11:52:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=iAIdlmQi304G2VHxb5nA14xlg37hvOMH8tX4iTr+Lzw=; b=b4UtiHy7fBj56hp6XvF32xA/y43Qo0DxVj22NoigfyDeY9FBn8NN+BSSXYLL3hRPnf QKridnw94lFx21/SuQAgqLOiRJExZ7c3rWuJL4meQeWRCA1qhmw/M3II0+bfd2j9PVo4 vTImZZqn9aacGML0PaUmexOwJjbfi+AvDMs8s= Received: by 10.213.28.68 with SMTP id l4mr1201709ebc.58.1320346353497; Thu, 03 Nov 2011 11:52:33 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.102.19 with HTTP; Thu, 3 Nov 2011 11:52:13 -0700 (PDT) In-Reply-To: <376CEC01195C894CB9F8A3C274029A96B13E8463@FISH-EX2K10-01.azaleos.net> References: <376CEC01195C894CB9F8A3C274029A96B13E83B0@FISH-EX2K10-01.azaleos.net> <4eb2e1b0.c9cfe00a.2c67.49d7@mx.google.com> <376CEC01195C894CB9F8A3C274029A96B13E8463@FISH-EX2K10-01.azaleos.net> From: Jonathan Ellis Date: Thu, 3 Nov 2011 13:52:13 -0500 Message-ID: Subject: Re: Problem after upgrade to 1.0.1 To: user@cassandra.apache.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Does restarting the node fix this? On Thu, Nov 3, 2011 at 1:51 PM, Bryce Godfrey w= rote: > Disk utilization is actually about 80% higher than what is reported for > nodetool ring across all my nodes on the data drive > > > > Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation=A0| T: > 206.926.1978 | M: 206.849.2477 > > > > From: Dan Hendry [mailto:dan.hendry.junk@gmail.com] > Sent: Thursday, November 03, 2011 11:47 AM > To: user@cassandra.apache.org > Subject: RE: Problem after upgrade to 1.0.1 > > > > Regarding load growth, presumably you are referring to the load as report= ed > by JMX/nodetool. Have you actually looked at the disk utilization on the > nodes themselves? Potential issue I have seen: > http://www.mail-archive.com/user@cassandra.apache.org/msg18142.html > > > > Dan > > > > From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com] > Sent: November-03-11 14:40 > To: user@cassandra.apache.org > Subject: Problem after upgrade to 1.0.1 > > > > I recently upgraded from 0.8.6 to 1.0.1 and everything seemed to go just > fine with the rolling upgrade.=A0 But now I=92m having extreme load growt= h on > one of my nodes (and others are growing faster than usual also).=A0 I > attempted to run a cfstats against the extremely large node that was seei= ng > 2x the load of others and I get this error below.=A0 I=92m also went into= the > o.a.c.db.HintedHandoffManager mbean and attempted to list pending hints t= o > see if it was growing out of control for some reason, but that just times > out eventually for any node.=A0 I=92m not sure what to do next with this = issue. > > > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Column Family: HintsColumnFami= ly > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 SSTable count: 3 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Space used (live): 12681676= 437 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Space used (total): 1023313= 0272 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Number of Keys (estimate): = 384 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Memtable Columns Count: 117= 704 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Memtable Data Size: 1151073= 07 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Memtable Switch Count: 66 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Read Count: 0 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Read Latency: NaN ms. > > =A0=A0=A0=A0=A0 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0Write Count: 21203290 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Write Latency: 0.014 ms. > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Pending Tasks: 0 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Key cache capacity: 3 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Key cache size: 0 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Key cache hit rate: NaN > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Row cache: disabled > > =A0 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0Compacted row minimum size:= 30130993 > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Compacted row maximum size:= 9223372036854775807 > > Exception in thread "main" java.lang.IllegalStateException: Unable to > compute ceiling for max when histogram overflowed > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.jav= a:170) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:395) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyStor= e.java:293) > > =A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Nat= ive Method) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :39) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:25) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanI= ntrospector.java:93) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanI= ntrospector.java:27) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:= 208) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Defaul= tMBeanServerInterceptor.java:666) > > =A0=A0=A0=A0=A0=A0=A0 at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:6= 38) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionIm= pl.java:1404) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImp= l.java:72) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMI= ConnectionImpl.java:1265) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMICo= nnectionImpl.java:1360) > > =A0=A0=A0=A0=A0=A0=A0 at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionI= mpl.java:600) > > =A0=A0=A0=A0=A0=A0=A0 at sun.reflect.GeneratedMethodAccessor15.invoke(Unk= nown Source) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:25) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > > =A0=A0=A0=A0=A0=A0=A0 at sun.rmi.transport.Transport$1.run(Transport.java= :159) > > =A0=A0=A0=A0=A0=A0=A0 at java.security.AccessController.doPrivileged(Nati= ve Method) > > =A0=A0=A0=A0=A0=A0=A0 at sun.rmi.transport.Transport.serviceCall(Transpor= t.java:155) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.ja= va:790) > > =A0=A0=A0=A0=A0=A0=A0 at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.jav= a:649) > > =A0=A0=A0=A0=A0=A0=A0 at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886) > > =A0=A0=A0=A0=A0=A0=A0 at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:662) > > > > Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation=A0| T: > 206.926.1978 | M: 206.849.2477 > > > > No virus found in this incoming message. > Checked by AVG - www.avg.com > Version: 9.0.920 / Virus Database: 271.1.1/3993 - Release Date: 11/03/11 > 03:39:00 --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com