Nope. I did alter two of my own column families to use Leveled compaction and then ran scrub
on each node, is the only change I have made from the upgrade.
Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T: 206.926.1978 | M: 206.849.2477
-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com]
Sent: Thursday, November 03, 2011 11:44 AM
To: user@cassandra.apache.org
Subject: Re: Problem after upgrade to 1.0.1
Just to rule it out: you didn't do anything tricky like update HintsColumnFamily to use compression?
On Thu, Nov 3, 2011 at 1:39 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> I recently upgraded from 0.8.6 to 1.0.1 and everything seemed to go
> just fine with the rolling upgrade. But now I'm having extreme load
> growth on one of my nodes (and others are growing faster than usual
> also). I attempted to run a cfstats against the extremely large node
> that was seeing 2x the load of others and I get this error below. I'm
> also went into the o.a.c.db.HintedHandoffManager mbean and attempted
> to list pending hints to see if it was growing out of control for some
> reason, but that just times out eventually for any node. I'm not sure what to do next
with this issue.
>
>
>
> Column Family: HintsColumnFamily
>
> SSTable count: 3
>
> Space used (live): 12681676437
>
> Space used (total): 10233130272
>
> Number of Keys (estimate): 384
>
> Memtable Columns Count: 117704
>
> Memtable Data Size: 115107307
>
> Memtable Switch Count: 66
>
> Read Count: 0
>
> Read Latency: NaN ms.
>
> Write Count: 21203290
>
> Write Latency: 0.014 ms.
>
> Pending Tasks: 0
>
> Key cache capacity: 3
>
> Key cache size: 0
>
> Key cache hit rate: NaN
>
> Row cache: disabled
>
> Compacted row minimum size: 30130993
>
> Compacted row maximum size: 9223372036854775807
>
> Exception in thread "main" java.lang.IllegalStateException: Unable to
> compute ceiling for max when histogram overflowed
>
> at
> org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.
> java:170)
>
> at
> org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:39
> 5)
>
> at
> org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyS
> tore.java:293)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe
> anIntrospector.java:93)
>
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe
> anIntrospector.java:27)
>
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.ja
> va:208)
>
> at
> com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65
> )
>
> at
> com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:21
> 6)
>
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Def
> aultMBeanServerInterceptor.java:666)
>
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.jav
> a:638)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectio
> nImpl.java:1404)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnection
> Impl.java:72)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(
> RMIConnectionImpl.java:1265)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RM
> IConnectionImpl.java:1360)
>
> at
> javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnecti
> onImpl.java:600)
>
> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:53
> 5)
>
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport
> .java:790)
>
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.
> java:649)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
> tor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
>
>
> Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T:
> 206.926.1978 | M: 206.849.2477
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
|