cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daning Wang <dan...@netseer.com>
Subject Re: Cassandra Exception
Date Wed, 28 Mar 2012 18:18:53 GMT
We upgraded to 1.0.8, and looks the problem is gone.

Thanks for your help,

Daning

On Sun, Mar 25, 2012 at 9:54 AM, aaron morton <aaron@thelastpickle.com>wrote:

> Can you go to those nodes and run describe cluster ? Also check the logs
> on the machines that are marked as UNREACHABLE .
>
> A node will be marked as UNREACHABLE if it is DOWN or if it did not
> respond in time.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 23/03/2012, at 11:29 AM, Daning Wang wrote:
>
> Thanks Aaron. when I do describe cluster, always there are "UNREACHABLE",
> but nodetool ring is fine. it is pretty busy cluster, read 3K/sec
>
> $ cassandra-cli -h localhost -u root -pw cassy
> Connected to: "Production Cluster" on localhost/9160
> Welcome to the Cassandra CLI.
>
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
>
> [root@unknown] describe cluster;
> Cluster Information:
>    Snitch: org.apache.cassandra.locator.SimpleSnitch
>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>    Schema versions:
>     UNREACHABLE: [10.218.17.208, 10.123.83.114, 10.120.205.48,
> 10.240.113.171]
>     e331e720-4844-11e1-0000-d808570c0dfd: [10.28.78.123, 10.28.20.200,
> 10.6.115.239]
> [root@unknown]
>
> $ nodetool -h localhost ring
> Address         DC          Rack        Status State   Load
> Owns    Token
>
> 141784319550391026443072753096570088105
> 10.28.78.123    datacenter1 rack1       Up     Normal  5.46 GB
> 16.67%  0
> 10.120.205.48   datacenter1 rack1       Up     Normal  5.49 GB
> 16.67%  28356863910078205288614550619314017621
> 10.6.115.239    datacenter1 rack1       Up     Normal  5.53 GB
> 16.67%  56713727820156410577229101238628035242
> 10.28.20.200    datacenter1 rack1       Up     Normal  5.51 GB
> 16.67%  85070591730234615865843651857942052863
> 10.123.83.114   datacenter1 rack1       Up     Normal  5.49 GB
> 16.67%  113427455640312821154458202477256070484
> 10.240.113.171  datacenter1 rack1       Up     Normal  5.43 GB
> 16.67%  141784319550391026443072753096570088105
>
>
> Daning
>
>
> On Thu, Mar 22, 2012 at 1:47 PM, aaron morton <aaron@thelastpickle.com>wrote:
>
>>  java.io.IOError:
>>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>>> cfId=-387130991
>>>
>> Schema may have diverged between nodes.
>>
>> use cassandra-cli and run describe cluster; to see how many schema
>> versions you have.
>>
>> Cheers
>>
>>   -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 22/03/2012, at 6:27 AM, Daning Wang wrote:
>>
>> and we are on 0.8.6.
>>
>>
>>
>> On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <daning@netseer.com> wrote:
>>
>>> Hi All,
>>>
>>>
>>> We got lots of Exception in the log, and later the server crashed. any
>>> idea what is happening and how to fix it?
>>>
>>> ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482
>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>> Thread[RequestResponseStage:4,5,main]
>>> java.io.IOError: java.io.EOFException
>>>         at
>>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>>>         at
>>> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
>>>         at
>>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>>>         at
>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>         at java.lang.Thread.run(Thread.java:722)
>>> Caused by: java.io.EOFException
>>>         at java.io.DataInputStream.readFully(DataInputStream.java:197)
>>>         at
>>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
>>>         at
>>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>>>         at
>>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
>>>         ... 6 more
>>> ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480
>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>> Thread[RequestResponseStage:2,5,main]
>>> java.io.IOError:
>>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>>> cfId=-387130991
>>>         at
>>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>>>         at
>>> org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
>>>         at
>>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>>>         at
>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>         at java.lang.Thread.run(Thread.java:722)
>>> Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException:
>>> Couldn't find cfId=-387130991
>>>         at
>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
>>>         at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
>>>         at
>>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
>>>         at
>>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>>>         at
>>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
>>>
>>>
>>>
>>> The is the exception before server crashes.
>>>
>>>
>>> ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808
>>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>>> Thread[ReadRepairStage:299,5,main]
>>> java.lang.RuntimeException:
>>> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
>>> shut down
>>>         at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>         at java.lang.Thread.run(Thread.java:722)
>>> Caused by: java.util.concurrent.RejectedExecutionException:
>>> ThreadPoolExecutor has shut down
>>>         at
>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>>         at
>>> org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
>>>         at
>>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
>>>         at
>>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
>>>         at
>>> org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
>>>         at
>>> org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
>>>         at
>>> org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
>>>         at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>         ... 3 more
>>>
>>> Thank you in advance,
>>>
>>> Daning
>>>
>>
>>
>>
>
>

Mime
View raw message