incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: problem with bootstrap
Date Fri, 11 Mar 2011 21:31:27 GMT
IMHO creating a keyspace with RF higher than the number of nodes sounds like a bug. It puts
the cluster into a bad place. It may even be a regression, will take a look at the code.

The assertion is interesting. Can you reproduce it with logging at debug and post the results?
Could you try to reproduce it with a clean cluster?

Thanks
Aaron


On 11/03/2011, at 10:24 PM, Patrik Modesto <patrik.modesto@gmail.com> wrote:

> Unfortunately I can't provide the info, I deleted it. It was in wery
> strange state.
> 
> I started with new cluster today, 2 nodes, each with
> auto_bootstrap:true. I can create a keyspace with RF=3, but I can't
> insert any data in it. It didn't happen with the old cluster which
> made me think. How could I insert data in the old cluster in keyspace
> with RF=3 but with just 2 nodes? I found out that the cluster had 3
> nodes for short time in the past. We had to remove/return one node but
> that was enough for the cluster to accept writes to keyspace with RF=3
> even with just 2 nodes.
> 
> So I tried to recreate the cluster state:
> 
> I have 4 clean server, cassndra 0.7.3, auto_bootstrap:true
> 
> 1) setup & run node1 - success
> 
> 2) create keyspace Context with rf=3" and create CF Url via
> cassandra-cli - success
> 
> 3) list Url - Internal error processing get_range_slicesl
> node1:
> ERROR 09:46:28,725 Internal error processing get_range_slices
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (1)
> 
> 4) setup & run node2 - success
> 
> 5) list Url on node1 - Internal error processing get_range_slicesl
> node1:
> ERROR 09:46:28,725 Internal error processing get_range_slices
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (1)
> 
> 6) list Url on node2 - Internal error processing get_range_slicesl
> node2:
> ERROR 09:50:54,231 Internal error processing get_range_slices
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> 
> 7) insert on node1 - Internal error processing insert
> node1:
> ERROR 09:53:11,669 Internal error processing insert
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> 
> 8) insert on node2 - Internal error processing insert
> node2:
> ERROR 09:53:54,833 Internal error processing insert
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> 
> 9) setup & run node3 - success
> 
> 10) list Url on node1 - success
> 
> 11) insert in Url on node1 - success
> 
> 12) stop cassandra on node3 - success
> 
> 13) list & insert on node1&2 - success
> 
> 14) loadbalance on node1 - Exception in thread "main"
> java.lang.IllegalStateException: replication factor (3) exceeds number
> of endpoints (2)
> 
> 15) setup & run node4 - success
> 
> 16) list Url on node4 - success BUT
> node4:
> ERROR 10:05:38,452 Fatal exception in thread
> Thread[RequestResponseStage:1,5,main]
> java.lang.AssertionError
>        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
>        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> ERROR 10:05:38,462 Fatal exception in thread
> Thread[RequestResponseStage:17,5,main]
> java.lang.AssertionError
>        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
>        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> 
> 17) loadbalance on node1 - success
> 
> 18) list Url on node4 - success BUT
> node4:
> ERROR 10:09:58,251 Fatal exception in thread
> Thread[RequestResponseStage:18,5,main]
> java.lang.AssertionError
>        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
>        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> ERROR 10:09:58,257 Fatal exception in thread
> Thread[RequestResponseStage:5,5,main]
> java.lang.AssertionError
>        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
>        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> 
> 19) repair on node4 - after long long wait I killed it, non of the
> nodes report any error
> 
> 20) list Url on node1 - success BUT
> node1:
> ERROR 10:18:53,715 Fatal exception in thread
> Thread[RequestResponseStage:6,5,main]
> java.lang.AssertionError
>        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
>        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> 
> So, I can't get the cluster to the state it was before the reinstalled
> it, where I couldn't bootstrap new node. I hope it was just
> combination of cassandra upgrades and lots of scheme changes and that
> it won't happen in production. OTOH there is the AssertionError which
> doesn't look good but I can insert/retrieve the data.
> 
> Regards,
> Patrik

Mime
View raw message