cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daning Wang <dan...@netseer.com>
Subject Re: Cassandra Exception
Date Thu, 22 Mar 2012 22:29:47 GMT
Thanks Aaron. when I do describe cluster, always there are "UNREACHABLE",
but nodetool ring is fine. it is pretty busy cluster, read 3K/sec

$ cassandra-cli -h localhost -u root -pw cassy
Connected to: "Production Cluster" on localhost/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[root@unknown] describe cluster;
Cluster Information:
   Snitch: org.apache.cassandra.locator.SimpleSnitch
   Partitioner: org.apache.cassandra.dht.RandomPartitioner
   Schema versions:
    UNREACHABLE: [10.218.17.208, 10.123.83.114, 10.120.205.48,
10.240.113.171]
    e331e720-4844-11e1-0000-d808570c0dfd: [10.28.78.123, 10.28.20.200,
10.6.115.239]
[root@unknown]

$ nodetool -h localhost ring
Address         DC          Rack        Status State   Load
Owns    Token

141784319550391026443072753096570088105
10.28.78.123    datacenter1 rack1       Up     Normal  5.46 GB
16.67%  0
10.120.205.48   datacenter1 rack1       Up     Normal  5.49 GB
16.67%  28356863910078205288614550619314017621
10.6.115.239    datacenter1 rack1       Up     Normal  5.53 GB
16.67%  56713727820156410577229101238628035242
10.28.20.200    datacenter1 rack1       Up     Normal  5.51 GB
16.67%  85070591730234615865843651857942052863
10.123.83.114   datacenter1 rack1       Up     Normal  5.49 GB
16.67%  113427455640312821154458202477256070484
10.240.113.171  datacenter1 rack1       Up     Normal  5.43 GB
16.67%  141784319550391026443072753096570088105


Daning


On Thu, Mar 22, 2012 at 1:47 PM, aaron morton <aaron@thelastpickle.com>wrote:

> java.io.IOError:
>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>> cfId=-387130991
>>
> Schema may have diverged between nodes.
>
> use cassandra-cli and run describe cluster; to see how many schema
> versions you have.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/03/2012, at 6:27 AM, Daning Wang wrote:
>
> and we are on 0.8.6.
>
>
>
> On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <daning@netseer.com> wrote:
>
>> Hi All,
>>
>>
>> We got lots of Exception in the log, and later the server crashed. any
>> idea what is happening and how to fix it?
>>
>> ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[RequestResponseStage:4,5,main]
>> java.io.IOError: java.io.EOFException
>>         at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>>         at
>> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
>>         at
>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>>         at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>         at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.io.EOFException
>>         at java.io.DataInputStream.readFully(DataInputStream.java:197)
>>         at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
>>         at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>>         at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
>>         ... 6 more
>> ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[RequestResponseStage:2,5,main]
>> java.io.IOError:
>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>> cfId=-387130991
>>         at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>>         at
>> org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
>>         at
>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>>         at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>         at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException:
>> Couldn't find cfId=-387130991
>>         at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
>>         at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
>>         at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
>>         at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>>         at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
>>
>>
>>
>> The is the exception before server crashes.
>>
>>
>> ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[ReadRepairStage:299,5,main]
>> java.lang.RuntimeException:
>> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
>> shut down
>>         at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>         at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.util.concurrent.RejectedExecutionException:
>> ThreadPoolExecutor has shut down
>>         at
>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>         at
>> org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
>>         at
>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
>>         at
>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
>>         at
>> org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
>>         at
>> org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
>>         at
>> org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
>>         at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>         ... 3 more
>>
>> Thank you in advance,
>>
>> Daning
>>
>
>
>

Mime
View raw message