incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cory Mintz <cory.p.mi...@gmail.com>
Subject keyspace not copied to new node
Date Fri, 28 Dec 2012 19:48:25 GMT
I am trying to add a second node to a cluster that is currently a single
node, with a single key space on it.

* Cluster names are in sync
* They both have the same seed node (the first one)
* Both of the same snitch (Ec2Snitch)
* I am not filling in an initial_token on either

When I start the new node it joins the cluster and gets 50% of cluster.
Everything looks good, except the existing keyspace never is copied over
and the new node starts recording exceptions. This can also be seen in
nodetool that the Load has not been split.

Any help would be appreciated.

nodetool ring after join:
10.125.0.6      us-east     1a          Up     Normal  6.7 KB
50.02%              58390160951331053490088942307836377318
10.125.0.17     us-east     1a          Up     Normal  8.73 MB
49.98%              143434328038307359414303841008091364408

Log file for the existing node:
 INFO [GossipStage:1] 2012-12-28 18:40:23,945 Gossiper.java (line 848) Node
/10.125.0.6 has restarted, now UP
 INFO [GossipStage:1] 2012-12-28 18:40:23,946 Gossiper.java (line 816)
InetAddress /10.125.0.6 is now UP
 INFO [GossipStage:1] 2012-12-28 18:40:40,593 Gossiper.java (line 830)
InetAddress /10.125.0.6 is now dead.
 INFO [GossipTasks:1] 2012-12-28 18:41:09,292 Gossiper.java (line 644)
FatClient /10.125.0.6 has been silent for 30000ms, removing from gossip
 INFO [GossipStage:1] 2012-12-28 18:45:18,219 Gossiper.java (line 850) Node
/10.125.0.6 is now part of the cluster
 INFO [GossipStage:1] 2012-12-28 18:45:18,220 Gossiper.java (line 816)
InetAddress /10.125.0.6 is now UP
 INFO [GossipStage:1] 2012-12-28 18:46:17,471 ColumnFamilyStore.java (line
659) Enqueuing flush of Memtable-LocationInfo@782297898(35/43
serialized/live bytes, 1 ops)
 INFO [FlushWriter:6] 2012-12-28 18:46:17,471 Memtable.java (line 264)
Writing Memtable-LocationInfo@782297898(35/43 serialized/live bytes, 1 ops)
 INFO [FlushWriter:6] 2012-12-28 18:46:17,477 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-40-Data.db
(89 bytes) for commitlog position ReplayPosition(segmentId=1356717215209,
position=81403)
 INFO [CompactionExecutor:7] 2012-12-28 18:46:17,478 CompactionTask.java
(line 109) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-39-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-40-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-38-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-37-Data.db')]
 INFO [CompactionExecutor:7] 2012-12-28 18:46:17,554 CompactionTask.java
(line 221) Compacted to
[/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-41-Data.db,].
702 to 435 (~61% of original) bytes for 4 keys at 0.005531MB/s.  Time: 75ms.
 INFO [MemoryMeter:1] 2012-12-28 19:00:28,396 Memtable.java (line 213)
CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is
3.167859982557678 (just-counted was 3.167505874454515).  calculation took
24ms for 54 columns
 INFO [GossipStage:1] 2012-12-28 19:01:20,035 StorageService.java (line
1287) Removing token 58390160951331053490088942307836377318 for /10.125.0.6
 INFO [OptionalTasks:1] 2012-12-28 19:01:20,035 HintedHandOffManager.java
(line 180) Deleting any stored hints for /10.125.0.6
 INFO [OptionalTasks:1] 2012-12-28 19:01:20,037 ColumnFamilyStore.java
(line 659) Enqueuing flush of Memtable-HintsColumnFamily@614495718(23832/94370
serialized/live bytes, 73 ops)
 INFO [GossipStage:1] 2012-12-28 19:01:20,037 ColumnFamilyStore.java (line
659) Enqueuing flush of Memtable-LocationInfo@2119262016(35/43
serialized/live bytes, 1 ops)
 INFO [FlushWriter:7] 2012-12-28 19:01:20,038 Memtable.java (line 264)
Writing Memtable-HintsColumnFamily@614495718(23832/94370 serialized/live
bytes, 73 ops)
 INFO [FlushWriter:7] 2012-12-28 19:01:20,047 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-5-Data.db
(66 bytes) for commitlog position ReplayPosition(segmentId=1356717215209,
position=114728)
 INFO [FlushWriter:7] 2012-12-28 19:01:20,048 Memtable.java (line 264)
Writing Memtable-LocationInfo@2119262016(35/43 serialized/live bytes, 1 ops)
 INFO [CompactionExecutor:8] 2012-12-28 19:01:20,051 CompactionTask.java
(line 109) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-5-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-4-Data.db')]
 INFO [FlushWriter:7] 2012-12-28 19:01:20,052 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-42-Data.db
(89 bytes) for commitlog position ReplayPosition(segmentId=1356717215209,
position=114728)
 INFO [CompactionExecutor:8] 2012-12-28 19:01:20,059 CompactionTask.java
(line 221) Compacted to
[/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-6-Data.db,].
132 to 66 (~50% of original) bytes for 1 keys at 0.012589MB/s.  Time: 5ms.

Log file for the new node:
 INFO [main] 2012-12-28 18:45:08,211 AbstractCassandraDaemon.java (line
101) Logging initialized
 INFO [main] 2012-12-28 18:45:08,216 AbstractCassandraDaemon.java (line
122) JVM vendor/version: OpenJDK 64-Bit Server VM/1.6.0_24
 INFO [main] 2012-12-28 18:45:08,217 AbstractCassandraDaemon.java (line
123) Heap size: 855638016/855638016
 INFO [main] 2012-12-28 18:45:08,222 AbstractCassandraDaemon.java (line
124) Classpath:
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.7.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.7.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
 INFO [main] 2012-12-28 18:45:10,045 CLibrary.java (line 111) JNA mlockall
successful
 INFO [main] 2012-12-28 18:45:10,055 DatabaseDescriptor.java (line 123)
Loading settings from file:/etc/cassandra/cassandra.yaml
 INFO [main] 2012-12-28 18:45:10,472 DatabaseDescriptor.java (line 182)
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
 INFO [main] 2012-12-28 18:45:11,055 DatabaseDescriptor.java (line 246)
Global memtable threshold is enabled at 272MB
 INFO [main] 2012-12-28 18:45:11,107 Ec2Snitch.java (line 63) EC2Snitch
using region: us-east, zone: 1a.
 INFO [main] 2012-12-28 18:45:11,972 CacheService.java (line 96)
Initializing key cache with capacity of 40 MBs.
 INFO [main] 2012-12-28 18:45:11,985 CacheService.java (line 107)
Scheduling key cache save to each 14400 seconds (going to save all keys).
 INFO [main] 2012-12-28 18:45:11,987 CacheService.java (line 121)
Initializing row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
 INFO [main] 2012-12-28 18:45:12,035 CacheService.java (line 133)
Scheduling row cache save to each 0 seconds (going to save all keys).
 INFO [main] 2012-12-28 18:45:12,413 DatabaseDescriptor.java (line 509)
Couldn't detect any schema definitions in local storage.
 INFO [main] 2012-12-28 18:45:12,413 DatabaseDescriptor.java (line 512)
Found table data in data directories. Consider using the CLI to define your
schema.
 INFO [main] 2012-12-28 18:45:12,468 CommitLog.java (line 124) No commitlog
files found; skipping replay
 INFO [main] 2012-12-28 18:45:12,518 StorageService.java (line 424)
Cassandra version: 1.1.7
 INFO [main] 2012-12-28 18:45:12,518 StorageService.java (line 425) Thrift
API version: 19.33.0
 INFO [main] 2012-12-28 18:45:12,554 StorageService.java (line 426) CQL
supported versions: 2.0.0,3.0.0-beta1 (default: 2.0.0)
 INFO [main] 2012-12-28 18:45:12,643 StorageService.java (line 456) Loading
persisted ring state
 INFO [main] 2012-12-28 18:45:12,646 StorageService.java (line 537)
Starting up server gossip
 INFO [main] 2012-12-28 18:45:12,661 ColumnFamilyStore.java (line 659)
Enqueuing flush of Memtable-LocationInfo@157438507(126/157 serialized/live
bytes, 3 ops)
 INFO [FlushWriter:1] 2012-12-28 18:45:12,661 Memtable.java (line 264)
Writing Memtable-LocationInfo@157438507(126/157 serialized/live bytes, 3
ops)
 INFO [FlushWriter:1] 2012-12-28 18:45:12,744 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.db
(234 bytes) for commitlog position ReplayPosition(segmentId=1356720312385,
position=586)
 INFO [main] 2012-12-28 18:45:12,756 Ec2Snitch.java (line 113) Ec2Snitch
adding ApplicationState ec2region=us-east ec2zone=1a
 INFO [main] 2012-12-28 18:45:12,792 MessagingService.java (line 284)
Starting Messaging Service on port 7000
 INFO [main] 2012-12-28 18:45:12,803 StorageService.java (line 775)
JOINING: waiting for ring information
 INFO [GossipStage:1] 2012-12-28 18:45:12,932 Gossiper.java (line 851) Node
/10.125.0.17 is now part of the cluster
 INFO [GossipStage:1] 2012-12-28 18:45:12,932 Gossiper.java (line 817)
InetAddress /10.125.0.17 is now UP
 INFO [GossipStage:1] 2012-12-28 18:45:12,936 ColumnFamilyStore.java (line
659) Enqueuing flush of Memtable-LocationInfo@1281435610(52/65
serialized/live bytes, 2 ops)
 INFO [FlushWriter:1] 2012-12-28 18:45:12,937 Memtable.java (line 264)
Writing Memtable-LocationInfo@1281435610(52/65 serialized/live bytes, 2 ops)
 INFO [FlushWriter:1] 2012-12-28 18:45:12,946 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db
(165 bytes) for commitlog position ReplayPosition(segmentId=1356720312385,
position=769)
 INFO [main] 2012-12-28 18:45:42,815 StorageService.java (line 775)
JOINING: schema complete, ready to bootstrap
 INFO [main] 2012-12-28 18:45:42,816 StorageService.java (line 775)
JOINING: getting bootstrap token
 INFO [main] 2012-12-28 18:45:42,836 BootStrapper.java (line 128) New token
will be 58390160951331053490088942307836377318 to assume load from /
10.125.0.17
 INFO [main] 2012-12-28 18:45:42,838 ColumnFamilyStore.java (line 659)
Enqueuing flush of Memtable-LocationInfo@617013740(36/45 serialized/live
bytes, 1 ops)
 INFO [FlushWriter:1] 2012-12-28 18:45:42,838 Memtable.java (line 264)
Writing Memtable-LocationInfo@617013740(36/45 serialized/live bytes, 1 ops)
 INFO [FlushWriter:1] 2012-12-28 18:45:42,843 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db
(87 bytes) for commitlog position ReplayPosition(segmentId=1356720312385,
position=865)
 INFO [main] 2012-12-28 18:45:42,845 StorageService.java (line 775)
JOINING: sleeping 30000 ms for pending range setup
 INFO [main] 2012-12-28 18:46:12,845 StorageService.java (line 775)
JOINING: Starting to bootstrap...
 INFO [main] 2012-12-28 18:46:12,862 ColumnFamilyStore.java (line 659)
Enqueuing flush of Memtable-LocationInfo@1563763559(53/66 serialized/live
bytes, 2 ops)
 INFO [FlushWriter:1] 2012-12-28 18:46:12,863 Memtable.java (line 264)
Writing Memtable-LocationInfo@1563763559(53/66 serialized/live bytes, 2 ops)
 INFO [FlushWriter:1] 2012-12-28 18:46:12,869 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db
(163 bytes) for commitlog position ReplayPosition(segmentId=1356720312385,
position=1046)
 INFO [main] 2012-12-28 18:46:12,910 StorageService.java (line 1120) Node /
10.125.0.6 state jump to normal
 INFO [main] 2012-12-28 18:46:12,911 StorageService.java (line 688)
Bootstrap/Replace/Move completed! Now serving reads.
 INFO [CompactionExecutor:1] 2012-12-28 18:46:12,955 CompactionTask.java
(line 109) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db')]
 INFO [main] 2012-12-28 18:46:13,017 CassandraDaemon.java (line 124)
Binding thrift service to /0.0.0.0:9160
 INFO [main] 2012-12-28 18:46:13,021 CassandraDaemon.java (line 133) Using
TFastFramedTransport with a max frame size of 15728640 bytes.
 INFO [main] 2012-12-28 18:46:13,053 CassandraDaemon.java (line 160) Using
synchronous/threadpool thrift server on /0.0.0.0 : 9160
 INFO [CompactionExecutor:1] 2012-12-28 18:46:13,057 CompactionTask.java
(line 221) Compacted to
[/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db,].
649 to 435 (~67% of original) bytes for 4 keys at 0.004148MB/s.  Time:
100ms.
 INFO [Thread-5] 2012-12-28 18:46:13,060 CassandraDaemon.java (line 212)
Listening for thrift clients...
ERROR [MutationStage:33] 2012-12-28 18:48:40,085
RowMutationVerbHandler.java (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=1000
    at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
    at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
    at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
    at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
    at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
    at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
ERROR [MutationStage:34] 2012-12-28 18:48:40,609
RowMutationVerbHandler.java (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=1000
    at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
    at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
    at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
    at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
    at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
    at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
ERROR [MutationStage:35] 2012-12-28 18:48:41,154
RowMutationVerbHandler.java (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=1000
    at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
    at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
    at
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
    at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
    at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
    at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)

Mime
View raw message