cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Hiramoto <k...@hiramoto.org>
Subject Re: balancing load
Date Mon, 17 Jan 2011 18:20:08 GMT
On 01/17/11 15:54, Edward Capriolo wrote:
> Just to head the next possible problem. If you run 'nodetool cleanup'
> on each node and some of your nodes still have more data then others,
> then it probably means your are writing the majority of data to a few
> keys. ( you probably do not want to do that )
>
> If that happens, you can use nodetool cfstats on each node and ensure
> that the 'max row compacted size' is roughly the same on all nodes. If
> you have one or two really big rows that could explain your imbalance.


Well, I did a lengthy repair/cleanup  on each node.  but still have data
mainly on two nodes (I have RF=2)
 ./apache-cassandra-0.7.0/bin/nodetool --host host3 ring
Address         Status State   Load            Owns   
Token                                      
                                                      
119098828422328462212181112601118874007    
10.1.4.10     Up     Normal  347.11 MB       30.00% 
0                                          
10.1.4.12     Up     Normal  49.41 KB        20.00% 
34028236692093846346337460743176821145     
10.1.4.13     Up     Normal  54.35 KB        20.00% 
68056473384187692692674921486353642290     
10.1.4.15     Up     Normal  19.09 GB        16.21% 
95643579558861158157614918209686336369     
10.1.4.14     Up     Normal  15.62 GB        13.79% 
119098828422328462212181112601118874007


in "cfstats" i see:
Compacted row minimum size: 1131752
Compacted row maximum size: 8582860529
Compacted row mean size: 1402350749

on the lowest used node i see:
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0

I basicly have  MyKeyspace.Offer[UID] = value    my "value"  is at most
500 bytes.

UID i just use a 12 random alpha numeric values  [A-Z][0-9]

Should i try and adjust my tokens to fix the imbalance or something else?

I'm using Redhat EL  5.5

java -version
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.5) (rhel-1.16.b17.el5-x86_64)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)

I have some errors in my logs:

ERROR [ReadStage:1747] 2011-01-17 18:13:53,988
DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor
java.lang.AssertionError
        at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:178)
        at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:132)
        at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:70)
        at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59)
        at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
        at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1215)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1107)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1077)
        at org.apache.cassandra.db.Table.getRow(Table.java:384)
        at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60)
        at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)
        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
ERROR [ReadStage:1747] 2011-01-17 18:13:53,989
AbstractCassandraDaemon.java (line 91) Fatal exception in thread
Thread[ReadStage:1747,5,main]
java.lang.AssertionError
        at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:178)
        at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:132)
        at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:70)
        at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59)
        at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
        at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1215)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1107)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1077)
        at org.apache.cassandra.db.Table.getRow(Table.java:384)
        at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60)
        at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)
        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)



Thanks,

Karl


Mime
View raw message