incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe <watche...@gmail.com>
Subject Re: SSTables not opened on new cluste
Date Fri, 03 May 2013 20:47:23 GMT
Unfortunately not, I've moved on to trying to add the nodes the current
cluster and then decommission the "old" ones.

But even that is not working, this is the strangest of things : while
trying to add a new node, I
 - set its token to  an existing value+1
 - ensure the yaml (clutser name, partitionner, etc) are the same
 - verified I can connect to the :7000 ports across machines
 - cleared the data and commitlog directory

when I start the node, it goes through the bootstrap process but never
"imports" the schema from the cluster (the data/ directory stays empty,
permissions are correct) and I get errors when some reads come in after the
bootstrap completes. I've tried restarting the node with -D replace.token
The log is below. I've been at it all day so I've been adding the node &
decommissioning it, assuming that would clear any state in the cluster.

 INFO [main] 2013-05-03 21:10:22,126 StorageService.java (line 788)
JOINING: waiting for ring information

 INFO [GossipStage:1] 2013-05-03 21:10:26,421 Gossiper.java (line 838)
InetAddress /172.16.0.42 is now dead.

 INFO [GossipStage:1] 2013-05-03 21:10:26,427 StorageService.java (line
1303) Removing token Token(bytes[03abaa3000000000000001]) for /172.16.0.42

 INFO [GossipStage:1] 2013-05-03 21:10:26,429 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@444462602(47/58
serialized/live bytes, 2 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,430 Memtable.java (line 264)
Writing Memtable-LocationInfo@444462602(47/58 serialized/live bytes, 2 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,542 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db
(160 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=769)

 INFO [GossipStage:1] 2013-05-03 21:10:26,544 Gossiper.java (line 858) Node
/{other_node_A} is now part of the cluster

 INFO [GossipStage:1] 2013-05-03 21:10:26,545 Gossiper.java (line 824)
InetAddress /{other_node_A} is now UP

 INFO [GossipStage:1] 2013-05-03 21:10:26,547 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@1290155526(30/37
serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,548 Memtable.java (line 264)
Writing Memtable-LocationInfo@1290155526(30/37 serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,662 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=862)

 INFO [GossipStage:1] 2013-05-03 21:10:26,664 Gossiper.java (line 858) Node
/{other_node_B} is now part of the cluster

 INFO [GossipStage:1] 2013-05-03 21:10:26,665 Gossiper.java (line 824)
InetAddress /{other_node_B} is now UP

 INFO [GossipStage:1] 2013-05-03 21:10:26,666 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@899933215(30/37
serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,667 Memtable.java (line 264)
Writing Memtable-LocationInfo@899933215(30/37 serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,779 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=955)

 INFO [GossipStage:1] 2013-05-03 21:10:26,781 Gossiper.java (line 858) Node
/{other_node_C} is now part of the cluster

 INFO [GossipStage:1] 2013-05-03 21:10:26,782 Gossiper.java (line 824)
InetAddress /{other_node_C} is now UP

 INFO [GossipStage:1] 2013-05-03 21:10:26,784 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@1542631196(30/37
serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:26,785 Memtable.java (line 264)
Writing Memtable-LocationInfo@1542631196(30/37 serialized/live bytes, 1 ops)

 INFO [CompactionExecutor:4] 2013-05-03 21:10:26,789 CompactionTask.java
(line 107) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.db')]

 INFO [FlushWriter:1] 2013-05-03 21:10:26,939 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=1048)

 INFO [GossipStage:1] 2013-05-03 21:10:26,942 Gossiper.java (line 838)
InetAddress /172.16.0.39 is now dead.

 INFO [CompactionExecutor:4] 2013-05-03 21:10:26,986 CompactionTask.java
(line 232) Compacted to
[/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db,].
566 to 458 (~80% of original) bytes for 4 keys at 0.002287MB/s.  Time:
191ms.

 INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
JOINING: schema complete

 INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
JOINING: waiting for pending range calculation

 INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
JOINING: calculation complete, ready to bootstrap

 INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
JOINING: getting bootstrap token

 INFO [main] 2013-05-03 21:10:52,135 ColumnFamilyStore.java (line 674)
Enqueuing flush of Memtable-LocationInfo@1692466348(31/38 serialized/live
bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:52,136 Memtable.java (line 264)
Writing Memtable-LocationInfo@1692466348(31/38 serialized/live bytes, 1 ops)

 INFO [FlushWriter:1] 2013-05-03 21:10:52,252 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db
(82 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=1139)

 INFO [main] 2013-05-03 21:10:52,254 StorageService.java (line 788)
JOINING: sleeping 30000 ms for pending range setup

 INFO [main] 2013-05-03 21:11:22,255 StorageService.java (line 788)
JOINING: Starting to bootstrap...

 INFO [main] 2013-05-03 21:11:22,272 ColumnFamilyStore.java (line 674)
Enqueuing flush of Memtable-LocationInfo@1458621244(48/60 serialized/live
bytes, 2 ops)

 INFO [FlushWriter:1] 2013-05-03 21:11:22,273 Memtable.java (line 264)
Writing Memtable-LocationInfo@1458621244(48/60 serialized/live bytes, 2 ops)

 INFO [FlushWriter:1] 2013-05-03 21:11:22,386 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db
(158 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=1315)

 INFO [CompactionExecutor:8] 2013-05-03 21:11:22,389 CompactionTask.java
(line 107) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db')]

 INFO [main] 2013-05-03 21:11:22,396 StorageService.java (line 1133) Node /
5.135.140.99 state jump to normal

 INFO [main] 2013-05-03 21:11:22,397 StorageService.java (line 701)
Bootstrap/Replace/Move completed! Now serving reads.

 INFO [main] 2013-05-03 21:11:22,415 CassandraDaemon.java (line 125)
Binding thrift service to /0.0.0.0:9160

 INFO [main] 2013-05-03 21:11:22,417 CassandraDaemon.java (line 134) Using
TFastFramedTransport with a max frame size of 15728640 bytes.

 INFO [main] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 161) Using
synchronous/threadpool thrift server on /0.0.0.0 : 9160

 INFO [Thread-9] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 213)
Listening for thrift clients...

 INFO [CompactionExecutor:8] 2013-05-03 21:11:22,542 CompactionTask.java
(line 232) Compacted to
[/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-9-Data.db,].
782 to 489 (~62% of original) bytes for 4 keys at 0.003088MB/s.  Time:
151ms.

Mime
View raw message