I also encounter the schema disagreement in my 0.8.1 cluster today…

The disagreement occurs when I create a column family using the hector api, and I found the following errors in my cassandra/system.log

ERROR [pool-2-thread-99] 2011-08-03 11:21:18,051 Cassandra.java (line 3378) Internal error processing remove
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
at org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:360)
at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:241)
at org.apache.cassandra.service.StorageProxy.access$000(StorageProxy.java:62)
at org.apache.cassandra.service.StorageProxy$1.apply(StorageProxy.java:99)
at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:210)
at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:154)
at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:560)
at org.apache.cassandra.thrift.CassandraServer.internal_remove(CassandraServer.java:539)
at org.apache.cassandra.thrift.CassandraServer.remove(CassandraServer.java:547)
at org.apache.cassandra.thrift.Cassandra$Processor$remove.process(Cassandra.java:3370)
at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

And when I try to decommission, I got this:

ERROR [pool-2-thread-90] 2011-08-03 11:24:35,611 Cassandra.java (line 3462) Internal error processing batch_mutate
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
at org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:360)
at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:241)
at org.apache.cassandra.service.StorageProxy.access$000(StorageProxy.java:62)
at org.apache.cassandra.service.StorageProxy$1.apply(StorageProxy.java:99)
at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:210)
at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:154)
at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:560)
at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:511)
at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:519)
at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

What does this mean? 

Thanks.

-- 
Dikang Gu
0086 - 18611140205

On Tuesday, August 2, 2011 at 6:04 PM, aaron morton wrote:

Hang on, using brain now. 

That is triggering a small bug in the code see https://issues.apache.org/jira/browse/CASSANDRA-2984

For not just remove the column meta data. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton

On 2 Aug 2011, at 21:19, aaron morton wrote:

What do you see when you run describe cluster; in the cassandra-cli ? Whats the exact error you get and is there anything in the server side logs ?

Have you added other CF's before adding this one ? Did the schema agree before starting this statement?

I ran the statement below on the current trunk and it worked. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton

On 2 Aug 2011, at 12:08, Dikang Gu wrote:

I thought the schema disagree problem was already solved in 0.8.1...

On possible solution is to decommission the disagree node and rejoin it.


On Tue, Aug 2, 2011 at 8:01 AM, Yi Yang <yyang@me.com> wrote:
Dear all,

I'm always meeting mp with schema disagree problems while trying to create a column family like this, using cassandra-cli:

create column family sd
   with column_type = 'Super'
   and key_validation_class = 'UUIDType'
   and comparator = 'LongType'
   and subcomparator = 'UTF8Type'
   and column_metadata = [
       {
       column_name: 'time',
       validation_class : 'LongType'
       },{
       column_name: 'open',
       validation_class : 'FloatType'
       },{
       column_name: 'high',
       validation_class : 'FloatType'
       },{
       column_name: 'low',
       validation_class : 'FloatType'
       },{
       column_name: 'close',
       validation_class : 'FloatType'
       },{
       column_name: 'volumn',
       validation_class : 'LongType'
       },{
       column_name: 'splitopen',
       validation_class : 'FloatType'
       },{
       column_name: 'splithigh',
       validation_class : 'FloatType'
       },{
       column_name: 'splitlow',
       validation_class : 'FloatType'
       },{
       column_name: 'splitclose',
       validation_class : 'FloatType'
       },{
       column_name: 'splitvolume',
       validation_class : 'LongType'
       },{
       column_name: 'splitclose',
       validation_class : 'FloatType'
       }
   ]
;

I've tried to erase everything and restart Cassandra but this still happens.   But when I clear the column_metadata section this no more disagreement error.   Do you have any idea why this happens?

Environment: 2 VMs, using the same harddrive, Cassandra 0.8.1, Ubuntu 10.04
This is for testing only.   We'll move to dedicated servers later.

Best regards,
Yi



--
Dikang Gu

0086 - 18611140205