incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra Exception
Date Sun, 25 Mar 2012 16:54:05 GMT
Can you go to those nodes and run describe cluster ? Also check the logs on the machines that
are marked as UNREACHABLE . 

A node will be marked as UNREACHABLE if it is DOWN or if it did not respond in time. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 23/03/2012, at 11:29 AM, Daning Wang wrote:

> Thanks Aaron. when I do describe cluster, always there are "UNREACHABLE", but nodetool
ring is fine. it is pretty busy cluster, read 3K/sec
> 
> $ cassandra-cli -h localhost -u root -pw cassy
> Connected to: "Production Cluster" on localhost/9160
> Welcome to the Cassandra CLI.
> 
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
> 
> [root@unknown] describe cluster;
> Cluster Information:
>    Snitch: org.apache.cassandra.locator.SimpleSnitch
>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>    Schema versions: 
>     UNREACHABLE: [10.218.17.208, 10.123.83.114, 10.120.205.48, 10.240.113.171]
>     e331e720-4844-11e1-0000-d808570c0dfd: [10.28.78.123, 10.28.20.200, 10.6.115.239]
> [root@unknown] 
> 
> $ nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token
                                      
>                                                                                141784319550391026443072753096570088105
    
> 10.28.78.123    datacenter1 rack1       Up     Normal  5.46 GB         16.67%  0    
                                      
> 10.120.205.48   datacenter1 rack1       Up     Normal  5.49 GB         16.67%  28356863910078205288614550619314017621
     
> 10.6.115.239    datacenter1 rack1       Up     Normal  5.53 GB         16.67%  56713727820156410577229101238628035242
     
> 10.28.20.200    datacenter1 rack1       Up     Normal  5.51 GB         16.67%  85070591730234615865843651857942052863
     
> 10.123.83.114   datacenter1 rack1       Up     Normal  5.49 GB         16.67%  113427455640312821154458202477256070484
    
> 10.240.113.171  datacenter1 rack1       Up     Normal  5.43 GB         16.67%  141784319550391026443072753096570088105
  
> 
> 
> Daning
> 
> 
> On Thu, Mar 22, 2012 at 1:47 PM, aaron morton <aaron@thelastpickle.com> wrote:
>> java.io.IOError: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't
find cfId=-387130991
> Schema may have diverged between nodes. 
> 
> use cassandra-cli and run describe cluster; to see how many schema versions you have.
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 22/03/2012, at 6:27 AM, Daning Wang wrote:
> 
>> and we are on 0.8.6.
>> 
>> 
>> 
>> On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <daning@netseer.com> wrote:
>> Hi All, 
>> 
>> 
>> We got lots of Exception in the log, and later the server crashed. any idea what
is happening and how to fix it?
>> 
>> ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[RequestResponseStage:4,5,main]
>> java.io.IOError: java.io.EOFException
>>         at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>>         at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
>>         at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>         at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.io.EOFException
>>         at java.io.DataInputStream.readFully(DataInputStream.java:197)
>>         at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
>>         at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>>         at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
>>         ... 6 more
>> ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[RequestResponseStage:2,5,main]
>> java.io.IOError: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't
find cfId=-387130991
>>         at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>>         at org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
>>         at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>         at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't
find cfId=-387130991
>>         at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
>>         at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
>>         at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
>>         at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>>         at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
>> 
>> 
>> 
>> The is the exception before server crashes.
>> 
>> 
>> ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[ReadRepairStage:299,5,main]
>> java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor
has shut down
>>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>         at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
>>         at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>>         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>         at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>         at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
>>         at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
>>         at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
>>         at org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
>>         at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
>>         at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
>>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>         ... 3 more
>> 
>> Thank you in advance,
>> 
>> Daning
>> 
> 
> 


Mime
View raw message