cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: any ideas on what these mean
Date Mon, 01 Oct 2012 01:12:26 GMT
> ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java (line 134)
Exception in thread Thread[MigrationStage:1,5,main]
I *think* this is due to concurrent schema modifications. 
The found number is the one in the database for the matching KS / CF name, the second number
is the one in memory at the time. That's why the reboot fixed it.

If it happens again try  nodetool resetlocalschema  

> 2.  Is it affecting my data put?  (I have seen other weird validation exceptions where
my data is still put and I can read it from cassandra and I get no exception client side)
This write was dropped
> ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 RowMutationVerbHandler.java (line
61) Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740
I would make sure repair is running to resolve any inconsistencies created during the period
of wackyness.

> 3.  How do I reverse engineer what column familsy 13740 and 13739 are?  (ie. Their names?)
so I can check for data corruption.
On 1.1 X schema is stored in the schema_columnfamilies CF in t he system KS. In previous versions
it's in a different CF in the same KS. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 27/09/2012, at 4:04 AM, "Hiller, Dean" <Dean.Hiller@nrel.gov> wrote:

> We were consistently getting this exception over and over as we put data into the system.
 A reboot caused it to go away but we don't want to be rebooting in the future….
> 
> 1.  When does this occur?
> 2.  Is it affecting my data put?  (I have seen other weird validation exceptions where
my data is still put and I can read it from cassandra and I get no exception client side)
> 3.  How do I reverse engineer what column familsy 13740 and 13739 are?  (ie. Their names?)
so I can check for data corruption.
> 
> ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java (line 134)
Exception in thread Thread[MigrationStage:1,5,main]
> java.lang.RuntimeException: java.io.IOException: org.apache.cassandra.config.ConfigurationException:
Column family ID mismatch (found 13740; expected 13739)
>        at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Column
family ID mismatch (found 13740; expected 13739)
>        at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:676)
>        at org.apache.cassandra.db.DefsTable.updateColumnFamily(DefsTable.java:463)
>        at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:407)
>        at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:271)
>        at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:249)
>        at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:48)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 6 more
> Caused by: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch
(found 13740; expected 13739)
>        at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:698)
>        at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:672)        ...
12 more
> 
> We also see this exception in the same log which is ironic considering the above one
says found 13740!!!!!! And this one says couldn't find it….
> 
> ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 RowMutationVerbHandler.java (line
61) Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740
>        at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>        at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>        at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>        at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>        at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)


Mime
View raw message