cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryce Godfrey <Bryce.Godf...@azaleos.com>
Subject RE: Problem after upgrade to 1.0.1
Date Thu, 03 Nov 2011 18:47:45 GMT
Nope.  I did alter two of my own column families to use Leveled compaction and then ran scrub
on each node, is the only change I have made from the upgrade.

Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T: 206.926.1978 | M: 206.849.2477

-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com] 
Sent: Thursday, November 03, 2011 11:44 AM
To: user@cassandra.apache.org
Subject: Re: Problem after upgrade to 1.0.1

Just to rule it out: you didn't do anything tricky like update HintsColumnFamily to use compression?

On Thu, Nov 3, 2011 at 1:39 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> I recently upgraded from 0.8.6 to 1.0.1 and everything seemed to go 
> just fine with the rolling upgrade.  But now I'm having extreme load 
> growth on one of my nodes (and others are growing faster than usual 
> also).  I attempted to run a cfstats against the extremely large node 
> that was seeing 2x the load of others and I get this error below.  I'm 
> also went into the o.a.c.db.HintedHandoffManager mbean and attempted 
> to list pending hints to see if it was growing out of control for some 
> reason, but that just times out eventually for any node.  I'm not sure what to do next
with this issue.
>
>
>
>                Column Family: HintsColumnFamily
>
>                 SSTable count: 3
>
>                 Space used (live): 12681676437
>
>                 Space used (total): 10233130272
>
>                 Number of Keys (estimate): 384
>
>                 Memtable Columns Count: 117704
>
>                 Memtable Data Size: 115107307
>
>                 Memtable Switch Count: 66
>
>                 Read Count: 0
>
>                 Read Latency: NaN ms.
>
>                 Write Count: 21203290
>
>                 Write Latency: 0.014 ms.
>
>                 Pending Tasks: 0
>
>                 Key cache capacity: 3
>
>                 Key cache size: 0
>
>                 Key cache hit rate: NaN
>
>                 Row cache: disabled
>
>                 Compacted row minimum size: 30130993
>
>                 Compacted row maximum size: 9223372036854775807
>
> Exception in thread "main" java.lang.IllegalStateException: Unable to 
> compute ceiling for max when histogram overflowed
>
>         at
> org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.
> java:170)
>
>         at
> org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:39
> 5)
>
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyS
> tore.java:293)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:39)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:25)
>
>         at java.lang.reflect.Method.invoke(Method.java:597)
>
>         at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe
> anIntrospector.java:93)
>
>         at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe
> anIntrospector.java:27)
>
>         at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.ja
> va:208)
>
>         at
> com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65
> )
>
>         at
> com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:21
> 6)
>
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Def
> aultMBeanServerInterceptor.java:666)
>
>         at
> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.jav
> a:638)
>
>         at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectio
> nImpl.java:1404)
>
>         at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnection
> Impl.java:72)
>
>         at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(
> RMIConnectionImpl.java:1265)
>
>         at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RM
> IConnectionImpl.java:1360)
>
>         at
> javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnecti
> onImpl.java:600)
>
>         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown 
> Source)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:25)
>
>         at java.lang.reflect.Method.invoke(Method.java:597)
>
>         at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>
>         at sun.rmi.transport.Transport$1.run(Transport.java:159)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>
>         at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:53
> 5)
>
>         at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport
> .java:790)
>
>         at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.
> java:649)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
> tor.java:886)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:908)
>
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
> Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T:
> 206.926.1978 | M: 206.849.2477
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com

Mime
View raw message