incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramesh Natarajan <rames...@gmail.com>
Subject Re: nodetool cfstats on 1.0.0-rc1 throws an exception
Date Wed, 05 Oct 2011 17:39:50 GMT
I don't have access to the test system anymore. We did move to lower
number of CFs and dont see this problem any more.
I remember when I noticed the size in system.log it was little more
than UINT_MAX (4294967295). I was able to recreate it multiple times.
So I am wondering if there are any stats counters in the system which
is set to unsigned int instead of unsigned long?

thanks
Ramesh

On Tue, Oct 4, 2011 at 3:20 AM, aaron morton <aaron@thelastpickle.com> wrote:
> That row has a size of 819 peta bytes, so something is odd there. The error
> is a result of that value been so huge. When you rant he same script on
> 0.8.6 what was the max size of the Migrations CF ?
> As Jonathan says, it's unlikely anyone would have tested creating 5000 CF's.
> Most people only create a few 10's of CF's at most.
> either use fewer CF's or…
> * dump the Migrations CF using sstable2json to take a look around
> * work out steps to reproduce and report it on Jira
> Hope that helps.
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> On 4/10/2011, at 11:30 AM, Ramesh Natarajan wrote:
>
> We recreated the schema using the same input file on both clusters and they
> are running identical load.
> Isn't the exception thrown in the system CF?
> this line looks strange:
> Compacted row maximum size: 9223372036854775807
> thanks
> Ramesh
>
> On Mon, Oct 3, 2011 at 5:26 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> Looks like you have unexpectedly large rows in your 1.0 cluster but
>> not 0.8.  I guess you could use sstable2json to manually check your
>> row sizes.
>>
>> On Mon, Oct 3, 2011 at 5:20 PM, Ramesh Natarajan <ramesh25@gmail.com>
>> wrote:
>> > It happens all the time on 1.0. It doesn't happen on 0.8.6.  Is there
>> > any
>> > thing I can do to check?
>> > thanks
>> > Ramesh
>> >
>> > On Mon, Oct 3, 2011 at 5:15 PM, Jonathan Ellis <jbellis@gmail.com>
>> > wrote:
>> >>
>> >> My suspicion would be that it has more to do with "rare case when
>> >> running with 5000 CFs" than "1.0 regression."
>> >>
>> >> On Mon, Oct 3, 2011 at 5:00 PM, Ramesh Natarajan <ramesh25@gmail.com>
>> >> wrote:
>> >> > We have about 5000 column family and when we run the nodetool cfstats
>> >> > it
>> >> > throws out this exception...  this is running 1.0.0-rc1
>> >> > This seems to work on 0.8.6.  Is this a bug in 1.0.0?
>> >> >
>> >> > thanks
>> >> > Ramesh
>> >> > Keyspace: system
>> >> >         Read Count: 28
>> >> >         Read Latency: 5.8675 ms.
>> >> >         Write Count: 3
>> >> >         Write Latency: 0.166 ms.
>> >> >         Pending Tasks: 0
>> >> >                 Column Family: Schema
>> >> >                 SSTable count: 4
>> >> >                 Space used (live): 4293758276
>> >> >                 Space used (total): 4293758276
>> >> >                 Number of Keys (estimate): 5376
>> >> >                 Memtable Columns Count: 0
>> >> >                 Memtable Data Size: 0
>> >> >                 Memtable Switch Count: 0
>> >> >                 Read Count: 3
>> >> >                 Read Latency: NaN ms.
>> >> >                 Write Count: 0
>> >> >                 Write Latency: NaN ms.
>> >> >                 Pending Tasks: 0
>> >> >                 Key cache capacity: 53
>> >> >                 Key cache size: 2
>> >> >                 Key cache hit rate: NaN
>> >> >                 Row cache: disabled
>> >> >                 Compacted row minimum size: 104
>> >> >                 Compacted row maximum size: 1955666
>> >> >                 Compacted row mean size: 1508515
>> >> >                 Column Family: HintsColumnFamily
>> >> >                 SSTable count: 0
>> >> >                 Space used (live): 0
>> >> >                 Space used (total): 0
>> >> >                 Number of Keys (estimate): 0
>> >> >                 Memtable Columns Count: 0
>> >> >                 Memtable Data Size: 0
>> >> >                 Memtable Switch Count: 0
>> >> >                 Read Count: 5
>> >> >                 Read Latency: NaN ms.
>> >> >                 Write Count: 0
>> >> >                 Write Latency: NaN ms.
>> >> >                 Pending Tasks: 0
>> >> >                 Key cache capacity: 1
>> >> >                 Key cache size: 0
>> >> >                 Key cache hit rate: NaN
>> >> >                 Row cache: disabled
>> >> >                 Compacted row minimum size: 0
>> >> >                 Compacted row maximum size: 0
>> >> >                 Compacted row mean size: 0
>> >> >                 Column Family: LocationInfo
>> >> >                 SSTable count: 1
>> >> >                 Space used (live): 6947
>> >> >                 Space used (total): 6947
>> >> >                 Number of Keys (estimate): 128
>> >> >                 Memtable Columns Count: 0
>> >> >                 Memtable Data Size: 0
>> >> >                 Memtable Switch Count: 2
>> >> >                 Read Count: 20
>> >> >                 Read Latency: NaN ms.
>> >> >                 Write Count: 3
>> >> >                 Write Latency: NaN ms.
>> >> >                 Pending Tasks: 0
>> >> >                 Key cache capacity: 1
>> >> >                 Key cache size: 1
>> >> >                 Key cache hit rate: NaN
>> >> >                 Row cache: disabled
>> >> >                 Compacted row minimum size: 73
>> >> >                 Compacted row maximum size: 258
>> >> >                 Compacted row mean size: 185
>> >> >                 Column Family: Migrations
>> >> >                 SSTable count: 4
>> >> >                 Space used (live): 4315909643
>> >> >                 Space used (total): 4315909643
>> >> >                 Number of Keys (estimate): 512
>> >> >                 Memtable Columns Count: 0
>> >> >                 Memtable Data Size: 0
>> >> >                 Memtable Switch Count: 0
>> >> >                 Read Count: 0
>> >> >                 Read Latency: NaN ms.
>> >> >                 Write Count: 0
>> >> >                 Write Latency: NaN ms.
>> >> >                 Pending Tasks: 0
>> >> >                 Key cache capacity: 5
>> >> >                 Key cache size: 0
>> >> >                 Key cache hit rate: NaN
>> >> >                 Row cache: disabled
>> >> >                 Compacted row minimum size: 5839589
>> >> >                 Compacted row maximum size: 9223372036854775807
>> >> > Exception in thread "main" java.lang.IllegalStateException: Unable
to
>> >> > compute ceiling for max when histogram overflowed
>> >> >         at
>> >> >
>> >> >
>> >> > org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:170)
>> >> >         at
>> >> >
>> >> > org.apache.cassandra.db.DataTracker.getMeanRowSize(DataTracker.java:395)
>> >> >         at
>> >> >
>> >> >
>> >> > org.apache.cassandra.db.ColumnFamilyStore.getMeanRowSize(ColumnFamilyStore.java:275)
>> >> >         at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown
>> >> > Source)
>> >> >         at
>> >> >
>> >> >
>> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> >         at java.lang.reflect.Method.invoke(Method.java:597)
>> >> >         at
>> >> >
>> >> >
>> >> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>> >> >         at
>> >> >
>> >> >
>> >> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>> >> >         at
>> >> >
>> >> >
>> >> > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>> >> >         at
>> >> >
>> >> > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
>> >> >         at
>> >> >
>> >> > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
>> >> >         at
>> >> >
>> >> >
>> >> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
>> >> >         at
>> >> >
>> >> >
>> >> > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
>> >> >         at
>> >> >
>> >> >
>> >> > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
>> >> >         at
>> >> >
>> >> >
>> >> > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>> >> >         at
>> >> >
>> >> >
>> >> > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>> >> >         at
>> >> >
>> >> >
>> >> > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>> >> >         at
>> >> >
>> >> >
>> >> > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
>> >> >         at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown
>> >> > Source)
>> >> >         at
>> >> >
>> >> >
>> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> >         at java.lang.reflect.Method.invoke(Method.java:597)
>> >> >         at
>> >> > sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>> >> >         at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> >> >         at java.security.AccessController.doPrivileged(Native Method)
>> >> >         at
>> >> > sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> >> >         at
>> >> >
>> >> > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>> >> >         at
>> >> >
>> >> >
>> >> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>> >> >         at
>> >> >
>> >> >
>> >> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>> >> >         at
>> >> >
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> >         at
>> >> >
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> >         at java.lang.Thread.run(Thread.java:662)
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
>

Mime
View raw message