cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edmond Lau <>
Subject on bootstrapping a node
Date Wed, 28 Oct 2009 04:10:52 GMT
I'd like to improve my mental model of how Cassandra bootstrapping
works.  My understanding is that bootstrapping is just an extra step
during a node's startup where the node copies data from neighboring
nodes that, according to its token, it should own; afterwards, the
node behaves like any other node.

If that's correct, I have a few follow-up questions:

- At what point does the new node get inserted into the hash ring so
that reads/writes for keys get directed to it?
- What are the semantics of bootstrapping a node that's been in the
cluster before and already has some data that's possibly outdated?
Should this work?  This might be useful if a node's been out of
commission for sufficiently long period of time.
- If we pick a poor set of initial tokens, would it be sensible to
modify the tokens on existing nodes and then restart them with
bootstrapping in order to rebalance?

I've also noticed that I can get my cassandra cluster into a weird
state via bootstrapping, where it stops accepting reads/writes.  I'm
on Cassandra 0.4.1.  A simple repro case is to start all 3 nodes of a
3 node cluster (replication factor of 2) using bootstrapping.  Getting
a key that I've inserted then leads to an IndexOutOfBoundsException.
Another IndexOutOfBoundsException was thrown later while flushing.

DEBUG [pool-1-thread-2] 2009-10-28 02:18:19,907
(line 258) get
DEBUG [pool-1-thread-2] 2009-10-28 02:18:19,908
(line 307) multiget
ERROR [pool-1-thread-2] 2009-10-28 02:18:19,912 (line
647) Internal error processing get
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.RangeCheck(
        at java.util.ArrayList.get(
        at org.apache.cassandra.locator.RackUnawareStrategy.getStorageTokens(
        at org.apache.cassandra.locator.RackUnawareStrategy.getReadStorageEndPoints(
        at org.apache.cassandra.locator.RackUnawareStrategy.getReadStorageEndPoints(
        at org.apache.cassandra.service.StorageService.getReadStorageEndPoints(
        at org.apache.cassandra.service.StorageProxy.readProtocol(
        at org.apache.cassandra.service.CassandraServer.readColumnFamily(
        at org.apache.cassandra.service.CassandraServer.multigetColumns(
        at org.apache.cassandra.service.CassandraServer.multigetInternal(
        at org.apache.cassandra.service.CassandraServer.multiget(
        at org.apache.cassandra.service.CassandraServer.get(
        at org.apache.cassandra.service.Cassandra$Processor$get.process(
        at org.apache.cassandra.service.Cassandra$Processor.process(
        at org.apache.thrift.server.TThreadPoolServer$
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
        at java.util.concurrent.ThreadPoolExecutor$
 INFO [PERIODIC-FLUSHER-POOL:1] 2009-10-28 02:18:45,719 (line 369) LocationInfo has reached its
threshold; switching in a fresh Memtable
 INFO [PERIODIC-FLUSHER-POOL:1] 2009-10-28 02:18:45,720 (line 1178) Enqueuing flush of
 INFO [MEMTABLE-FLUSHER-POOL:1] 2009-10-28 02:18:45,721
(line 186) Flushing Memtable(LocationInfo)@1048641931
DEBUG [COMMIT-LOG-WRITER] 2009-10-28 02:18:45,877 (line
466) discard completed log segments for
6696265813.log', position=423), column family 4. CFIDs are Keyspace1:
TableMetadata(Standard2: 1, Super1: 0, Standard1: 2, StandardByUUID1:
3, }), system: TableMetadata(Locatio\
nInfo: 4, HintsColumnFamily: 5, }), Analytics: TableMetadata(total: 6,
domain: 7, movie: 8, provider: 9, country: 10, }), }
DEBUG [COMMIT-LOG-WRITER] 2009-10-28 02:18:45,878 (line
509) Marking replay position 423 on commit log
 INFO [MEMTABLE-FLUSHER-POOL:1] 2009-10-28 02:18:45,878
(line 220) Completed flushing
DEBUG [BOOT-STRAPPER:1] 2009-10-28 02:18:45,954
(line 83) Exception was generated at : 10/28/2009 02:18:45 on thread
java.lang.ArrayIndexOutOfBoundsException: -1
        at java.util.ArrayList.get(
        at org.apache.cassandra.service.StorageService.getAllRanges(
        at org.apache.cassandra.dht.BootStrapper.getRangesWithSourceTarget(
        at java.util.concurrent.Executors$
        at java.util.concurrent.FutureTask$Sync.innerRun(
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
        at java.util.concurrent.ThreadPoolExecutor$


View raw message