incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrik Modesto <patrik.mode...@gmail.com>
Subject Re: problem with bootstrap
Date Fri, 11 Mar 2011 09:24:52 GMT
Unfortunately I can't provide the info, I deleted it. It was in wery
strange state.

I started with new cluster today, 2 nodes, each with
auto_bootstrap:true. I can create a keyspace with RF=3, but I can't
insert any data in it. It didn't happen with the old cluster which
made me think. How could I insert data in the old cluster in keyspace
with RF=3 but with just 2 nodes? I found out that the cluster had 3
nodes for short time in the past. We had to remove/return one node but
that was enough for the cluster to accept writes to keyspace with RF=3
even with just 2 nodes.

So I tried to recreate the cluster state:

I have 4 clean server, cassndra 0.7.3, auto_bootstrap:true

1) setup & run node1 - success

2) create keyspace Context with rf=3" and create CF Url via
cassandra-cli - success

3) list Url - Internal error processing get_range_slicesl
node1:
ERROR 09:46:28,725 Internal error processing get_range_slices
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (1)

4) setup & run node2 - success

5) list Url on node1 - Internal error processing get_range_slicesl
node1:
ERROR 09:46:28,725 Internal error processing get_range_slices
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (1)

6) list Url on node2 - Internal error processing get_range_slicesl
node2:
ERROR 09:50:54,231 Internal error processing get_range_slices
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)

7) insert on node1 - Internal error processing insert
node1:
ERROR 09:53:11,669 Internal error processing insert
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)

8) insert on node2 - Internal error processing insert
node2:
ERROR 09:53:54,833 Internal error processing insert
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)

9) setup & run node3 - success

10) list Url on node1 - success

11) insert in Url on node1 - success

12) stop cassandra on node3 - success

13) list & insert on node1&2 - success

14) loadbalance on node1 - Exception in thread "main"
java.lang.IllegalStateException: replication factor (3) exceeds number
of endpoints (2)

15) setup & run node4 - success

16) list Url on node4 - success BUT
node4:
ERROR 10:05:38,452 Fatal exception in thread
Thread[RequestResponseStage:1,5,main]
java.lang.AssertionError
        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
ERROR 10:05:38,462 Fatal exception in thread
Thread[RequestResponseStage:17,5,main]
java.lang.AssertionError
        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

17) loadbalance on node1 - success

18) list Url on node4 - success BUT
node4:
ERROR 10:09:58,251 Fatal exception in thread
Thread[RequestResponseStage:18,5,main]
java.lang.AssertionError
        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
ERROR 10:09:58,257 Fatal exception in thread
Thread[RequestResponseStage:5,5,main]
java.lang.AssertionError
        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

19) repair on node4 - after long long wait I killed it, non of the
nodes report any error

20) list Url on node1 - success BUT
node1:
ERROR 10:18:53,715 Fatal exception in thread
Thread[RequestResponseStage:6,5,main]
java.lang.AssertionError
        at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
        at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

So, I can't get the cluster to the state it was before the reinstalled
it, where I couldn't bootstrap new node. I hope it was just
combination of cassandra upgrades and lots of scheme changes and that
it won't happen in production. OTOH there is the AssertionError which
doesn't look good but I can insert/retrieve the data.

Regards,
Patrik

Mime
View raw message