incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Problem after upgrade to 1.0.1
Date Fri, 04 Nov 2011 00:05:42 GMT
I found the problem and posted a patch on
https://issues.apache.org/jira/browse/CASSANDRA-3451.  If you build
with that patch and rerun scrub the exception should go away.

On Thu, Nov 3, 2011 at 2:08 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
> A restart fixed the load numbers, they are back to where I expect them to be now, but
disk utilization is double the load #.  I'm also still get the cfstats exception from any
node.
>
> -----Original Message-----
> From: Jonathan Ellis [mailto:jbellis@gmail.com]
> Sent: Thursday, November 03, 2011 11:52 AM
> To: user@cassandra.apache.org
> Subject: Re: Problem after upgrade to 1.0.1
>
> Does restarting the node fix this?
>
> On Thu, Nov 3, 2011 at 1:51 PM, Bryce Godfrey <Bryce.Godfrey@azaleos.com> wrote:
>> Disk utilization is actually about 80% higher than what is reported
>> for nodetool ring across all my nodes on the data drive
>>
>>
>>
>> Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T:
>> 206.926.1978 | M: 206.849.2477
>>
>>
>>
>> From: Dan Hendry [mailto:dan.hendry.junk@gmail.com]
>> Sent: Thursday, November 03, 2011 11:47 AM
>> To: user@cassandra.apache.org
>> Subject: RE: Problem after upgrade to 1.0.1
>>
>>
>>
>> Regarding load growth, presumably you are referring to the load as
>> reported by JMX/nodetool. Have you actually looked at the disk
>> utilization on the nodes themselves? Potential issue I have seen:
>> http://www.mail-archive.com/user@cassandra.apache.org/msg18142.html
>>
>>
>>
>> Dan
>>
>>
>>
>> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
>> Sent: November-03-11 14:40
>> To: user@cassandra.apache.org
>> Subject: Problem after upgrade to 1.0.1
>>
>>
>>
>> I recently upgraded from 0.8.6 to 1.0.1 and everything seemed to go
>> just fine with the rolling upgrade.  But now I'm having extreme load
>> growth on one of my nodes (and others are growing faster than usual
>> also).  I attempted to run a cfstats against the extremely large node
>> that was seeing 2x the load of others and I get this error below.  I'm
>> also went into the o.a.c.db.HintedHandoffManager mbean and attempted
>> to list pending hints to see if it was growing out of control for some
>> reason, but that just times out eventually for any node.  I'm not sure what to do
next with this issue.
>>
>>
>>
>>                Column Family: HintsColumnFamily
>>
>>                 SSTable count: 3
>>
>>                 Space used (live): 12681676437
>>
>>                 Space used (total): 10233130272
>>
>>                 Number of Keys (estimate): 384
>>
>>                 Memtable Columns Count: 117704
>>
>>                 Memtable Data Size: 115107307
>>
>>                 Memtable Switch Count: 66
>>
>>                 Read Count: 0
>>
>>                 Read Latency: NaN ms.
>>
>>                 Write Count: 21203290
>>
>>                 Write Latency: 0.014 ms.
>>
>>                 Pending Tasks: 0
>>
>>                 Key cache capacity: 3
>>
>>                 Key cache size: 0
>>
>>                 Key cache hit rate: NaN
>>
>>                 Row cache: disabled
>>
>>                 Compacted row minimum size: 30130993
>>
>>                 Compacted row maximum size: 9223372036854775807
>>
>> Exception in thread "main" java.lang.IllegalStateException: Unable to
>> compute ceiling for max when histogram overflowed
>>
>>         at
>> org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.
>> java:170)
>>
>>         at
>> org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:39
>> 5)
>>
>>         at
>> org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyS
>> tore.java:293)
>>
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
>> ava:39)
>>
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
>> orImpl.java:25)
>>
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>
>>         at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe
>> anIntrospector.java:93)
>>
>>         at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe
>> anIntrospector.java:27)
>>
>>         at
>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.ja
>> va:208)
>>
>>         at
>> com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65
>> )
>>
>>         at
>> com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:21
>> 6)
>>
>>         at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Def
>> aultMBeanServerInterceptor.java:666)
>>
>>         at
>> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.jav
>> a:638)
>>
>>         at
>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectio
>> nImpl.java:1404)
>>
>>         at
>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnection
>> Impl.java:72)
>>
>>         at
>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(
>> RMIConnectionImpl.java:1265)
>>
>>         at
>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RM
>> IConnectionImpl.java:1360)
>>
>>         at
>> javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnecti
>> onImpl.java:600)
>>
>>         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
>> orImpl.java:25)
>>
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>
>>         at
>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>>
>>         at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>
>>         at java.security.AccessController.doPrivileged(Native Method)
>>
>>         at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>
>>         at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:53
>> 5)
>>
>>         at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport
>> .java:790)
>>
>>         at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.
>> java:649)
>>
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
>> tor.java:886)
>>
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>> java:908)
>>
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T:
>> 206.926.1978 | M: 206.849.2477
>>
>>
>>
>> No virus found in this incoming message.
>> Checked by AVG - www.avg.com
>> Version: 9.0.920 / Virus Database: 271.1.1/3993 - Release Date:
>> 11/03/11
>> 03:39:00
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message