Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0CA14E4C8 for ; Fri, 28 Dec 2012 19:48:55 +0000 (UTC) Received: (qmail 41236 invoked by uid 500); 28 Dec 2012 19:48:52 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 41213 invoked by uid 500); 28 Dec 2012 19:48:52 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 41203 invoked by uid 99); 28 Dec 2012 19:48:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Dec 2012 19:48:52 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of cory.p.mintz@gmail.com designates 209.85.223.173 as permitted sender) Received: from [209.85.223.173] (HELO mail-ie0-f173.google.com) (209.85.223.173) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Dec 2012 19:48:46 +0000 Received: by mail-ie0-f173.google.com with SMTP id e13so13159470iej.4 for ; Fri, 28 Dec 2012 11:48:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=lB9Zijnf/EUAL8qdP0eihoKaYyJA4lDM3e9sCCnIYiE=; b=ej21qrcGXVlbVKZqMPEpszuPl1j7Eelgb2bjIGqdCaW5ZXxdGYppcpxuPVm0VxUTzz kW8Odteb2/LwI6yOPKylNZu7Yc91/3am20nibxIXF8Tv8WGPZjmjUH3ZPEk1Jl2tjCDL 5aB6RmZX88wg1mEXdLWP4iauOMZ3M3Wr8Aye25vkFMj5Xl9Ztc3JdIhRx78/CFDi7DWH PUUIQhWFq5HZQ6zAFK/ohb6uskcyl9n3zC/VnFZST7D5QE/6NVlVGxUeIPf0SOL2K8jL VZEoCa7XoDNTuR8aPRz2P3b5RdrcqpPGJidCrTvL7YEsidUcp4DJutMIx2gcro3g+7HO klVA== MIME-Version: 1.0 Received: by 10.50.1.200 with SMTP id 8mr25351178igo.76.1356724105960; Fri, 28 Dec 2012 11:48:25 -0800 (PST) Received: by 10.64.68.101 with HTTP; Fri, 28 Dec 2012 11:48:25 -0800 (PST) Date: Fri, 28 Dec 2012 14:48:25 -0500 Message-ID: Subject: keyspace not copied to new node From: Cory Mintz To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=e89a8f502e0c64dcb904d1eef20a X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f502e0c64dcb904d1eef20a Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I am trying to add a second node to a cluster that is currently a single node, with a single key space on it. * Cluster names are in sync * They both have the same seed node (the first one) * Both of the same snitch (Ec2Snitch) * I am not filling in an initial_token on either When I start the new node it joins the cluster and gets 50% of cluster. Everything looks good, except the existing keyspace never is copied over and the new node starts recording exceptions. This can also be seen in nodetool that the Load has not been split. Any help would be appreciated. nodetool ring after join: 10.125.0.6 us-east 1a Up Normal 6.7 KB 50.02% 58390160951331053490088942307836377318 10.125.0.17 us-east 1a Up Normal 8.73 MB 49.98% 143434328038307359414303841008091364408 Log file for the existing node: INFO [GossipStage:1] 2012-12-28 18:40:23,945 Gossiper.java (line 848) Node /10.125.0.6 has restarted, now UP INFO [GossipStage:1] 2012-12-28 18:40:23,946 Gossiper.java (line 816) InetAddress /10.125.0.6 is now UP INFO [GossipStage:1] 2012-12-28 18:40:40,593 Gossiper.java (line 830) InetAddress /10.125.0.6 is now dead. INFO [GossipTasks:1] 2012-12-28 18:41:09,292 Gossiper.java (line 644) FatClient /10.125.0.6 has been silent for 30000ms, removing from gossip INFO [GossipStage:1] 2012-12-28 18:45:18,219 Gossiper.java (line 850) Node /10.125.0.6 is now part of the cluster INFO [GossipStage:1] 2012-12-28 18:45:18,220 Gossiper.java (line 816) InetAddress /10.125.0.6 is now UP INFO [GossipStage:1] 2012-12-28 18:46:17,471 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-LocationInfo@782297898(35/43 serialized/live bytes, 1 ops) INFO [FlushWriter:6] 2012-12-28 18:46:17,471 Memtable.java (line 264) Writing Memtable-LocationInfo@782297898(35/43 serialized/live bytes, 1 ops) INFO [FlushWriter:6] 2012-12-28 18:46:17,477 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-40-Data.= db (89 bytes) for commitlog position ReplayPosition(segmentId=3D1356717215209, position=3D81403) INFO [CompactionExecutor:7] 2012-12-28 18:46:17,478 CompactionTask.java (line 109) Compacting [SSTableReader(path=3D'/var/lib/cassandra/data/system/LocationInfo/system-L= ocationInfo-hf-39-Data.db'), SSTableReader(path=3D'/var/lib/cassandra/data/system/LocationInfo/system-Lo= cationInfo-hf-40-Data.db'), SSTableReader(path=3D'/var/lib/cassandra/data/system/LocationInfo/system-Lo= cationInfo-hf-38-Data.db'), SSTableReader(path=3D'/var/lib/cassandra/data/system/LocationInfo/system-Lo= cationInfo-hf-37-Data.db')] INFO [CompactionExecutor:7] 2012-12-28 18:46:17,554 CompactionTask.java (line 221) Compacted to [/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-41-Data= .db,]. 702 to 435 (~61% of original) bytes for 4 keys at 0.005531MB/s. Time: 75ms= . INFO [MemoryMeter:1] 2012-12-28 19:00:28,396 Memtable.java (line 213) CFS(Keyspace=3D'system', ColumnFamily=3D'HintsColumnFamily') liveRatio is 3.167859982557678 (just-counted was 3.167505874454515). calculation took 24ms for 54 columns INFO [GossipStage:1] 2012-12-28 19:01:20,035 StorageService.java (line 1287) Removing token 58390160951331053490088942307836377318 for /10.125.0.6 INFO [OptionalTasks:1] 2012-12-28 19:01:20,035 HintedHandOffManager.java (line 180) Deleting any stored hints for /10.125.0.6 INFO [OptionalTasks:1] 2012-12-28 19:01:20,037 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-HintsColumnFamily@614495718(23832/94= 370 serialized/live bytes, 73 ops) INFO [GossipStage:1] 2012-12-28 19:01:20,037 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-LocationInfo@2119262016(35/43 serialized/live bytes, 1 ops) INFO [FlushWriter:7] 2012-12-28 19:01:20,038 Memtable.java (line 264) Writing Memtable-HintsColumnFamily@614495718(23832/94370 serialized/live bytes, 73 ops) INFO [FlushWriter:7] 2012-12-28 19:01:20,047 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-h= f-5-Data.db (66 bytes) for commitlog position ReplayPosition(segmentId=3D1356717215209, position=3D114728) INFO [FlushWriter:7] 2012-12-28 19:01:20,048 Memtable.java (line 264) Writing Memtable-LocationInfo@2119262016(35/43 serialized/live bytes, 1 ops= ) INFO [CompactionExecutor:8] 2012-12-28 19:01:20,051 CompactionTask.java (line 109) Compacting [SSTableReader(path=3D'/var/lib/cassandra/data/system/HintsColumnFamily/sys= tem-HintsColumnFamily-hf-5-Data.db'), SSTableReader(path=3D'/var/lib/cassandra/data/system/HintsColumnFamily/syst= em-HintsColumnFamily-hf-4-Data.db')] INFO [FlushWriter:7] 2012-12-28 19:01:20,052 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-42-Data.= db (89 bytes) for commitlog position ReplayPosition(segmentId=3D1356717215209, position=3D114728) INFO [CompactionExecutor:8] 2012-12-28 19:01:20,059 CompactionTask.java (line 221) Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-= hf-6-Data.db,]. 132 to 66 (~50% of original) bytes for 1 keys at 0.012589MB/s. Time: 5ms. Log file for the new node: INFO [main] 2012-12-28 18:45:08,211 AbstractCassandraDaemon.java (line 101) Logging initialized INFO [main] 2012-12-28 18:45:08,216 AbstractCassandraDaemon.java (line 122) JVM vendor/version: OpenJDK 64-Bit Server VM/1.6.0_24 INFO [main] 2012-12-28 18:45:08,217 AbstractCassandraDaemon.java (line 123) Heap size: 855638016/855638016 INFO [main] 2012-12-28 18:45:08,222 AbstractCassandraDaemon.java (line 124) Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-= fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/= cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.= 2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/li= b/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-l= ru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/= high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.ja= r:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandr= a/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/c= assandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.j= ar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metri= cs-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/us= r/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-lo= g4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassa= ndra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:= /usr/share/cassandra/apache-cassandra-1.1.7.jar:/usr/share/cassandra/apache= -cassandra-thrift-1.1.7.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/= share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/shar= e/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar INFO [main] 2012-12-28 18:45:10,045 CLibrary.java (line 111) JNA mlockall successful INFO [main] 2012-12-28 18:45:10,055 DatabaseDescriptor.java (line 123) Loading settings from file:/etc/cassandra/cassandra.yaml INFO [main] 2012-12-28 18:45:10,472 DatabaseDescriptor.java (line 182) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO [main] 2012-12-28 18:45:11,055 DatabaseDescriptor.java (line 246) Global memtable threshold is enabled at 272MB INFO [main] 2012-12-28 18:45:11,107 Ec2Snitch.java (line 63) EC2Snitch using region: us-east, zone: 1a. INFO [main] 2012-12-28 18:45:11,972 CacheService.java (line 96) Initializing key cache with capacity of 40 MBs. INFO [main] 2012-12-28 18:45:11,985 CacheService.java (line 107) Scheduling key cache save to each 14400 seconds (going to save all keys). INFO [main] 2012-12-28 18:45:11,987 CacheService.java (line 121) Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider INFO [main] 2012-12-28 18:45:12,035 CacheService.java (line 133) Scheduling row cache save to each 0 seconds (going to save all keys). INFO [main] 2012-12-28 18:45:12,413 DatabaseDescriptor.java (line 509) Couldn't detect any schema definitions in local storage. INFO [main] 2012-12-28 18:45:12,413 DatabaseDescriptor.java (line 512) Found table data in data directories. Consider using the CLI to define your schema. INFO [main] 2012-12-28 18:45:12,468 CommitLog.java (line 124) No commitlog files found; skipping replay INFO [main] 2012-12-28 18:45:12,518 StorageService.java (line 424) Cassandra version: 1.1.7 INFO [main] 2012-12-28 18:45:12,518 StorageService.java (line 425) Thrift API version: 19.33.0 INFO [main] 2012-12-28 18:45:12,554 StorageService.java (line 426) CQL supported versions: 2.0.0,3.0.0-beta1 (default: 2.0.0) INFO [main] 2012-12-28 18:45:12,643 StorageService.java (line 456) Loading persisted ring state INFO [main] 2012-12-28 18:45:12,646 StorageService.java (line 537) Starting up server gossip INFO [main] 2012-12-28 18:45:12,661 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-LocationInfo@157438507(126/157 serialized/live bytes, 3 ops) INFO [FlushWriter:1] 2012-12-28 18:45:12,661 Memtable.java (line 264) Writing Memtable-LocationInfo@157438507(126/157 serialized/live bytes, 3 ops) INFO [FlushWriter:1] 2012-12-28 18:45:12,744 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.d= b (234 bytes) for commitlog position ReplayPosition(segmentId=3D1356720312385= , position=3D586) INFO [main] 2012-12-28 18:45:12,756 Ec2Snitch.java (line 113) Ec2Snitch adding ApplicationState ec2region=3Dus-east ec2zone=3D1a INFO [main] 2012-12-28 18:45:12,792 MessagingService.java (line 284) Starting Messaging Service on port 7000 INFO [main] 2012-12-28 18:45:12,803 StorageService.java (line 775) JOINING: waiting for ring information INFO [GossipStage:1] 2012-12-28 18:45:12,932 Gossiper.java (line 851) Node /10.125.0.17 is now part of the cluster INFO [GossipStage:1] 2012-12-28 18:45:12,932 Gossiper.java (line 817) InetAddress /10.125.0.17 is now UP INFO [GossipStage:1] 2012-12-28 18:45:12,936 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-LocationInfo@1281435610(52/65 serialized/live bytes, 2 ops) INFO [FlushWriter:1] 2012-12-28 18:45:12,937 Memtable.java (line 264) Writing Memtable-LocationInfo@1281435610(52/65 serialized/live bytes, 2 ops= ) INFO [FlushWriter:1] 2012-12-28 18:45:12,946 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.d= b (165 bytes) for commitlog position ReplayPosition(segmentId=3D1356720312385= , position=3D769) INFO [main] 2012-12-28 18:45:42,815 StorageService.java (line 775) JOINING: schema complete, ready to bootstrap INFO [main] 2012-12-28 18:45:42,816 StorageService.java (line 775) JOINING: getting bootstrap token INFO [main] 2012-12-28 18:45:42,836 BootStrapper.java (line 128) New token will be 58390160951331053490088942307836377318 to assume load from / 10.125.0.17 INFO [main] 2012-12-28 18:45:42,838 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-LocationInfo@617013740(36/45 serialized/live bytes, 1 ops) INFO [FlushWriter:1] 2012-12-28 18:45:42,838 Memtable.java (line 264) Writing Memtable-LocationInfo@617013740(36/45 serialized/live bytes, 1 ops) INFO [FlushWriter:1] 2012-12-28 18:45:42,843 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.d= b (87 bytes) for commitlog position ReplayPosition(segmentId=3D1356720312385, position=3D865) INFO [main] 2012-12-28 18:45:42,845 StorageService.java (line 775) JOINING: sleeping 30000 ms for pending range setup INFO [main] 2012-12-28 18:46:12,845 StorageService.java (line 775) JOINING: Starting to bootstrap... INFO [main] 2012-12-28 18:46:12,862 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-LocationInfo@1563763559(53/66 serialized/live bytes, 2 ops) INFO [FlushWriter:1] 2012-12-28 18:46:12,863 Memtable.java (line 264) Writing Memtable-LocationInfo@1563763559(53/66 serialized/live bytes, 2 ops= ) INFO [FlushWriter:1] 2012-12-28 18:46:12,869 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.d= b (163 bytes) for commitlog position ReplayPosition(segmentId=3D1356720312385= , position=3D1046) INFO [main] 2012-12-28 18:46:12,910 StorageService.java (line 1120) Node / 10.125.0.6 state jump to normal INFO [main] 2012-12-28 18:46:12,911 StorageService.java (line 688) Bootstrap/Replace/Move completed! Now serving reads. INFO [CompactionExecutor:1] 2012-12-28 18:46:12,955 CompactionTask.java (line 109) Compacting [SSTableReader(path=3D'/var/lib/cassandra/data/system/LocationInfo/system-L= ocationInfo-hf-4-Data.db'), SSTableReader(path=3D'/var/lib/cassandra/data/system/LocationInfo/system-Lo= cationInfo-hf-1-Data.db'), SSTableReader(path=3D'/var/lib/cassandra/data/system/LocationInfo/system-Lo= cationInfo-hf-2-Data.db'), SSTableReader(path=3D'/var/lib/cassandra/data/system/LocationInfo/system-Lo= cationInfo-hf-3-Data.db')] INFO [main] 2012-12-28 18:46:13,017 CassandraDaemon.java (line 124) Binding thrift service to /0.0.0.0:9160 INFO [main] 2012-12-28 18:46:13,021 CassandraDaemon.java (line 133) Using TFastFramedTransport with a max frame size of 15728640 bytes. INFO [main] 2012-12-28 18:46:13,053 CassandraDaemon.java (line 160) Using synchronous/threadpool thrift server on /0.0.0.0 : 9160 INFO [CompactionExecutor:1] 2012-12-28 18:46:13,057 CompactionTask.java (line 221) Compacted to [/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.= db,]. 649 to 435 (~67% of original) bytes for 4 keys at 0.004148MB/s. Time: 100ms. INFO [Thread-5] 2012-12-28 18:46:13,060 CassandraDaemon.java (line 212) Listening for thrift clients... ERROR [MutationStage:33] 2012-12-28 18:48:40,085 RowMutationVerbHandler.java (line 61) Error in row mutation org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=3D1000 at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySeri= alizer.java:126) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMu= tation.java:439) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMu= tation.java:447) at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395) at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandle= r.java:42) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:5= 9) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1= 110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 603) at java.lang.Thread.run(Thread.java:679) ERROR [MutationStage:34] 2012-12-28 18:48:40,609 RowMutationVerbHandler.java (line 61) Error in row mutation org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=3D1000 at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySeri= alizer.java:126) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMu= tation.java:439) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMu= tation.java:447) at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395) at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandle= r.java:42) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:5= 9) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1= 110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 603) at java.lang.Thread.run(Thread.java:679) ERROR [MutationStage:35] 2012-12-28 18:48:41,154 RowMutationVerbHandler.java (line 61) Error in row mutation org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=3D1000 at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySeri= alizer.java:126) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMu= tation.java:439) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMu= tation.java:447) at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395) at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandle= r.java:42) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:5= 9) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1= 110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 603) at java.lang.Thread.run(Thread.java:679) --e89a8f502e0c64dcb904d1eef20a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I am trying to add a second node to a cluster that is currently a single no= de, with a single key space on it.

* Cluster names are in sync
* = They both have the same seed node (the first one)
* Both of the same sni= tch (Ec2Snitch)
* I am not filling in an initial_token on either

When I start the ne= w node it joins the cluster and gets 50% of cluster. Everything looks good,= except the existing keyspace never is copied over and the new node starts = recording exceptions. This can also be seen in nodetool that the Load has n= ot been split.

Any help would be appreciated.

nodetool ring after join:
10.1= 25.0.6=A0=A0=A0=A0=A0 us-east=A0=A0=A0=A0 1a=A0=A0=A0=A0=A0=A0=A0=A0=A0 Up= =A0=A0=A0=A0 Normal=A0 6.7 KB=A0=A0=A0=A0=A0=A0=A0=A0=A0 50.02%=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0 58390160951331053490088942307836377318
10.12= 5.0.17=A0=A0=A0=A0 us-east=A0=A0=A0=A0 1a=A0=A0=A0=A0=A0=A0=A0=A0=A0 Up=A0= =A0=A0=A0 Normal=A0 8.73 MB=A0=A0=A0=A0=A0=A0=A0=A0 49.98%=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 143434328038307359414303841008091364408

Log file for the existing node:
=A0INFO [GossipStage:1] 2012-12-28 1= 8:40:23,945 Gossiper.java (line 848) Node /10= .125.0.6 has restarted, now UP
=A0INFO [GossipStage:1] 2012-12-28 18= :40:23,946 Gossiper.java (line 816) InetAddress /10.125.0.6 is now UP
=A0INFO [GossipStage:1] 2012-12-28 18:40:40,593 Gossiper.java (line 830) In= etAddress /10.125.0.6 is now dead.
=A0= INFO [GossipTasks:1] 2012-12-28 18:41:09,292 Gossiper.java (line 644) FatCl= ient /10.125.0.6 has been silent for 3000= 0ms, removing from gossip
=A0INFO [GossipStage:1] 2012-12-28 18:45:18,219 Gossiper.java (line 850) No= de /10.125.0.6 is now part of the cluster=
=A0INFO [GossipStage:1] 2012-12-28 18:45:18,220 Gossiper.java (line 816= ) InetAddress /10.125.0.6 is now UP
=A0INFO [GossipStage:1] 2012-12-28 18:46:17,471 ColumnFamilyStore.java (lin= e 659) Enqueuing flush of Memtable-LocationInfo@782297898(35/43 serialized/= live bytes, 1 ops)
=A0INFO [FlushWriter:6] 2012-12-28 18:46:17,471 Memta= ble.java (line 264) Writing Memtable-LocationInfo@782297898(35/43 serialize= d/live bytes, 1 ops)
=A0INFO [FlushWriter:6] 2012-12-28 18:46:17,477 Memtable.java (line 305) Co= mpleted flushing /var/lib/cassandra/data/system/LocationInfo/system-Locatio= nInfo-hf-40-Data.db (89 bytes) for commitlog position ReplayPosition(segmen= tId=3D1356717215209, position=3D81403)
=A0INFO [CompactionExecutor:7] 2012-12-28 18:46:17,478 CompactionTask.java = (line 109) Compacting [SSTableReader(path=3D'/var/lib/cassandra/data/sy= stem/LocationInfo/system-LocationInfo-hf-39-Data.db'), SSTableReader(pa= th=3D'/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-h= f-40-Data.db'), SSTableReader(path=3D'/var/lib/cassandra/data/syste= m/LocationInfo/system-LocationInfo-hf-38-Data.db'), SSTableReader(path= =3D'/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-= 37-Data.db')]
=A0INFO [CompactionExecutor:7] 2012-12-28 18:46:17,554 CompactionTask.java = (line 221) Compacted to [/var/lib/cassandra/data/system/LocationInfo/system= -LocationInfo-hf-41-Data.db,].=A0 702 to 435 (~61% of original) bytes for 4= keys at 0.005531MB/s.=A0 Time: 75ms.
=A0INFO [MemoryMeter:1] 2012-12-28 19:00:28,396 Memtable.java (line 213) CF= S(Keyspace=3D'system', ColumnFamily=3D'HintsColumnFamily') = liveRatio is 3.167859982557678 (just-counted was 3.167505874454515).=A0 cal= culation took 24ms for 54 columns
=A0INFO [GossipStage:1] 2012-12-28 19:01:20,035 StorageService.java (line 1= 287) Removing token 58390160951331053490088942307836377318 for /10.125.0.6
=A0INFO [OptionalTasks:1] 2012-12-28 1= 9:01:20,035 HintedHandOffManager.java (line 180) Deleting any stored hints = for /10.125.0.6
=A0INFO [OptionalTasks:1] 2012-12-28 19:01:20,037 ColumnFamilyStore.java (l= ine 659) Enqueuing flush of Memtable-HintsColumnFamily@614495718(23832/9437= 0 serialized/live bytes, 73 ops)
=A0INFO [GossipStage:1] 2012-12-28 19:0= 1:20,037 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Loca= tionInfo@2119262016(35/43 serialized/live bytes, 1 ops)
=A0INFO [FlushWriter:7] 2012-12-28 19:01:20,038 Memtable.java (line 264) Wr= iting Memtable-HintsColumnFamily@614495718(23832/94370 serialized/live byte= s, 73 ops)
=A0INFO [FlushWriter:7] 2012-12-28 19:01:20,047 Memtable.java= (line 305) Completed flushing /var/lib/cassandra/data/system/HintsColumnFa= mily/system-HintsColumnFamily-hf-5-Data.db (66 bytes) for commitlog positio= n ReplayPosition(segmentId=3D1356717215209, position=3D114728)
=A0INFO [FlushWriter:7] 2012-12-28 19:01:20,048 Memtable.java (line 264) Wr= iting Memtable-LocationInfo@2119262016(35/43 serialized/live bytes, 1 ops)<= br>=A0INFO [CompactionExecutor:8] 2012-12-28 19:01:20,051 CompactionTask.ja= va (line 109) Compacting [SSTableReader(path=3D'/var/lib/cassandra/data= /system/HintsColumnFamily/system-HintsColumnFamily-hf-5-Data.db'), SSTa= bleReader(path=3D'/var/lib/cassandra/data/system/HintsColumnFamily/syst= em-HintsColumnFamily-hf-4-Data.db')]
=A0INFO [FlushWriter:7] 2012-12-28 19:01:20,052 Memtable.java (line 305) Co= mpleted flushing /var/lib/cassandra/data/system/LocationInfo/system-Locatio= nInfo-hf-42-Data.db (89 bytes) for commitlog position ReplayPosition(segmen= tId=3D1356717215209, position=3D114728)
=A0INFO [CompactionExecutor:8] 2012-12-28 19:01:20,059 CompactionTask.java = (line 221) Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/s= ystem-HintsColumnFamily-hf-6-Data.db,].=A0 132 to 66 (~50% of original) byt= es for 1 keys at 0.012589MB/s.=A0 Time: 5ms.

Log file for the new node:
=A0INFO [main] 2012-12-28 18:45:08,211 Ab= stractCassandraDaemon.java (line 101) Logging initialized
=A0INFO [main]= 2012-12-28 18:45:08,216 AbstractCassandraDaemon.java (line 122) JVM vendor= /version: OpenJDK 64-Bit Server VM/1.6.0_24
=A0INFO [main] 2012-12-28 18:45:08,217 AbstractCassandraDaemon.java (line 1= 23) Heap size: 855638016/855638016
=A0INFO [main] 2012-12-28 18:45:08,22= 2 AbstractCassandraDaemon.java (line 124) Classpath: /usr/share/cassandra/l= ib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/c= assandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-= cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassa= ndra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.j= ar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/= cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.j= ar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra= /lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/= usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simp= le-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandr= a/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr= /share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/= slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/s= hare/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1= .0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/a= pache-cassandra-1.1.7.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.= 7.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress= .jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.= jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
=A0INFO [main] 2012-12-28 18:45:10,045 CLibrary.java (line 111) JNA mlockal= l successful
=A0INFO [main] 2012-12-28 18:45:10,055 DatabaseDescriptor.j= ava (line 123) Loading settings from file:/etc/cassandra/cassandra.yaml
=A0INFO [main] 2012-12-28 18:45:10,472 DatabaseDescriptor.java (line 182) D= iskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap=
=A0INFO [main] 2012-12-28 18:45:11,055 DatabaseDescriptor.java (line 24= 6) Global memtable threshold is enabled at 272MB
=A0INFO [main] 2012-12-28 18:45:11,107 Ec2Snitch.java (line 63) EC2Snitch u= sing region: us-east, zone: 1a.
=A0INFO [main] 2012-12-28 18:45:11,972 C= acheService.java (line 96) Initializing key cache with capacity of 40 MBs.<= br> =A0INFO [main] 2012-12-28 18:45:11,985 CacheService.java (line 107) Schedul= ing key cache save to each 14400 seconds (going to save all keys).
=A0IN= FO [main] 2012-12-28 18:45:11,987 CacheService.java (line 121) Initializing= row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.S= erializingCacheProvider
=A0INFO [main] 2012-12-28 18:45:12,035 CacheService.java (line 133) Schedul= ing row cache save to each 0 seconds (going to save all keys).
=A0INFO [= main] 2012-12-28 18:45:12,413 DatabaseDescriptor.java (line 509) Couldn'= ;t detect any schema definitions in local storage.
=A0INFO [main] 2012-12-28 18:45:12,413 DatabaseDescriptor.java (line 512) F= ound table data in data directories. Consider using the CLI to define your = schema.
=A0INFO [main] 2012-12-28 18:45:12,468 CommitLog.java (line 124)= No commitlog files found; skipping replay
=A0INFO [main] 2012-12-28 18:45:12,518 StorageService.java (line 424) Cassa= ndra version: 1.1.7
=A0INFO [main] 2012-12-28 18:45:12,518 StorageServic= e.java (line 425) Thrift API version: 19.33.0
=A0INFO [main] 2012-12-28 = 18:45:12,554 StorageService.java (line 426) CQL supported versions: 2.0.0,3= .0.0-beta1 (default: 2.0.0)
=A0INFO [main] 2012-12-28 18:45:12,643 StorageService.java (line 456) Loadi= ng persisted ring state
=A0INFO [main] 2012-12-28 18:45:12,646 StorageSe= rvice.java (line 537) Starting up server gossip
=A0INFO [main] 2012-12-2= 8 18:45:12,661 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtabl= e-LocationInfo@157438507(126/157 serialized/live bytes, 3 ops)
=A0INFO [FlushWriter:1] 2012-12-28 18:45:12,661 Memtable.java (line 264) Wr= iting Memtable-LocationInfo@157438507(126/157 serialized/live bytes, 3 ops)=
=A0INFO [FlushWriter:1] 2012-12-28 18:45:12,744 Memtable.java (line 305= ) Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-Loc= ationInfo-hf-1-Data.db (234 bytes) for commitlog position ReplayPosition(se= gmentId=3D1356720312385, position=3D586)
=A0INFO [main] 2012-12-28 18:45:12,756 Ec2Snitch.java (line 113) Ec2Snitch = adding ApplicationState ec2region=3Dus-east ec2zone=3D1a
=A0INFO [main] = 2012-12-28 18:45:12,792 MessagingService.java (line 284) Starting Messaging= Service on port 7000
=A0INFO [main] 2012-12-28 18:45:12,803 StorageService.java (line 775) JOINI= NG: waiting for ring information
=A0INFO [GossipStage:1] 2012-12-28 18:4= 5:12,932 Gossiper.java (line 851) Node /10.1= 25.0.17 is now part of the cluster
=A0INFO [GossipStage:1] 2012-12-28 18:45:12,932 Gossiper.java (line 817) In= etAddress /10.125.0.17 is now UP
=A0I= NFO [GossipStage:1] 2012-12-28 18:45:12,936 ColumnFamilyStore.java (line 65= 9) Enqueuing flush of Memtable-LocationInfo@1281435610(52/65 serialized/liv= e bytes, 2 ops)
=A0INFO [FlushWriter:1] 2012-12-28 18:45:12,937 Memtable.java (line 264) Wr= iting Memtable-LocationInfo@1281435610(52/65 serialized/live bytes, 2 ops)<= br>=A0INFO [FlushWriter:1] 2012-12-28 18:45:12,946 Memtable.java (line 305)= Completed flushing /var/lib/cassandra/data/system/LocationInfo/system-Loca= tionInfo-hf-2-Data.db (165 bytes) for commitlog position ReplayPosition(seg= mentId=3D1356720312385, position=3D769)
=A0INFO [main] 2012-12-28 18:45:42,815 StorageService.java (line 775) JOINI= NG: schema complete, ready to bootstrap
=A0INFO [main] 2012-12-28 18:45:= 42,816 StorageService.java (line 775) JOINING: getting bootstrap token
= =A0INFO [main] 2012-12-28 18:45:42,836 BootStrapper.java (line 128) New tok= en will be 58390160951331053490088942307836377318 to assume load from /10.125.0.17
=A0INFO [main] 2012-12-28 18:45:42,838 ColumnFamilyStore.java (line 659) En= queuing flush of Memtable-LocationInfo@617013740(36/45 serialized/live byte= s, 1 ops)
=A0INFO [FlushWriter:1] 2012-12-28 18:45:42,838 Memtable.java = (line 264) Writing Memtable-LocationInfo@617013740(36/45 serialized/live by= tes, 1 ops)
=A0INFO [FlushWriter:1] 2012-12-28 18:45:42,843 Memtable.java (line 305) Co= mpleted flushing /var/lib/cassandra/data/system/LocationInfo/system-Locatio= nInfo-hf-3-Data.db (87 bytes) for commitlog position ReplayPosition(segment= Id=3D1356720312385, position=3D865)
=A0INFO [main] 2012-12-28 18:45:42,845 StorageService.java (line 775) JOINI= NG: sleeping 30000 ms for pending range setup
=A0INFO [main] 2012-12-28 = 18:46:12,845 StorageService.java (line 775) JOINING: Starting to bootstrap.= ..
=A0INFO [main] 2012-12-28 18:46:12,862 ColumnFamilyStore.java (line 659) En= queuing flush of Memtable-LocationInfo@1563763559(53/66 serialized/live byt= es, 2 ops)
=A0INFO [FlushWriter:1] 2012-12-28 18:46:12,863 Memtable.java= (line 264) Writing Memtable-LocationInfo@1563763559(53/66 serialized/live = bytes, 2 ops)
=A0INFO [FlushWriter:1] 2012-12-28 18:46:12,869 Memtable.java (line 305) Co= mpleted flushing /var/lib/cassandra/data/system/LocationInfo/system-Locatio= nInfo-hf-4-Data.db (163 bytes) for commitlog position ReplayPosition(segmen= tId=3D1356720312385, position=3D1046)
=A0INFO [main] 2012-12-28 18:46:12,910 StorageService.java (line 1120) Node= /10.125.0.6 state jump to normal
=A0I= NFO [main] 2012-12-28 18:46:12,911 StorageService.java (line 688) Bootstrap= /Replace/Move completed! Now serving reads.
=A0INFO [CompactionExecutor:1] 2012-12-28 18:46:12,955 CompactionTask.java = (line 109) Compacting [SSTableReader(path=3D'/var/lib/cassandra/data/sy= stem/LocationInfo/system-LocationInfo-hf-4-Data.db'), SSTableReader(pat= h=3D'/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf= -1-Data.db'), SSTableReader(path=3D'/var/lib/cassandra/data/system/= LocationInfo/system-LocationInfo-hf-2-Data.db'), SSTableReader(path=3D&= #39;/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Da= ta.db')]
=A0INFO [main] 2012-12-28 18:46:13,017 CassandraDaemon.java (line 124) Bind= ing thrift service to /0.0.0.0:9160
= =A0INFO [main] 2012-12-28 18:46:13,021 CassandraDaemon.java (line 133) Usin= g TFastFramedTransport with a max frame size of 15728640 bytes.
=A0INFO [main] 2012-12-28 18:46:13,053 CassandraDaemon.java (line 160) Usin= g synchronous/threadpool thrift server on /0.0.0= .0 : 9160
=A0INFO [CompactionExecutor:1] 2012-12-28 18:46:13,057 Com= pactionTask.java (line 221) Compacted to [/var/lib/cassandra/data/system/Lo= cationInfo/system-LocationInfo-hf-5-Data.db,].=A0 649 to 435 (~67% of origi= nal) bytes for 4 keys at 0.004148MB/s.=A0 Time: 100ms.
=A0INFO [Thread-5] 2012-12-28 18:46:13,060 CassandraDaemon.java (line 212) = Listening for thrift clients...
ERROR [MutationStage:33] 2012-12-28 18:4= 8:40,085 RowMutationVerbHandler.java (line 61) Error in row mutation
org= .apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId= =3D1000
=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(Col= umnFamilySerializer.java:126)
=A0=A0=A0 at org.apache.cassandra.db.RowMu= tation$RowMutationSerializer.deserialize(RowMutation.java:439)
=A0=A0=A0= at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(R= owMutation.java:447)
=A0=A0=A0 at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java= :395)
=A0=A0=A0 at org.apache.cassandra.db.RowMutationVerbHandler.doVerb= (RowMutationVerbHandler.java:42)
=A0=A0=A0 at org.apache.cassandra.net.M= essageDeliveryTask.run(MessageDeliveryTask.java:59)
=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110)
=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$W= orker.run(ThreadPoolExecutor.java:603)
=A0=A0=A0 at java.lang.Thread.run= (Thread.java:679)
ERROR [MutationStage:34] 2012-12-28 18:48:40,609 RowMutationVerbHandler.jav= a (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnF= amilyException: Couldn't find cfId=3D1000
=A0=A0=A0 at org.apache.ca= ssandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:1= 26)
=A0=A0=A0 at org.apache.cassandra.db.RowMutation$RowMutationSerializer.dese= rialize(RowMutation.java:439)
=A0=A0=A0 at org.apache.cassandra.db.RowMu= tation$RowMutationSerializer.deserialize(RowMutation.java:447)
=A0=A0=A0= at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
=A0=A0=A0 at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutat= ionVerbHandler.java:42)
=A0=A0=A0 at org.apache.cassandra.net.MessageDel= iveryTask.run(MessageDeliveryTask.java:59)
=A0=A0=A0 at java.util.concur= rent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603)
=A0=A0=A0 at java.lang.Thread.run(Thread.java:679)
= ERROR [MutationStage:35] 2012-12-28 18:48:41,154 RowMutationVerbHandler.jav= a (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfI= d=3D1000
=A0=A0=A0 at org.apache.cassandra.db.ColumnFamilySerializer.des= erialize(ColumnFamilySerializer.java:126)
=A0=A0=A0 at org.apache.cassan= dra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)<= br> =A0=A0=A0 at org.apache.cassandra.db.RowMutation$RowMutationSerializer.dese= rialize(RowMutation.java:447)
=A0=A0=A0 at org.apache.cassandra.db.RowMu= tation.fromBytes(RowMutation.java:395)
=A0=A0=A0 at org.apache.cassandra= .db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
=A0=A0=A0 at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDelive= ryTask.java:59)
=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor.run= Worker(ThreadPoolExecutor.java:1110)
=A0=A0=A0 at java.util.concurrent.T= hreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
=A0=A0=A0 at java.lang.Thread.run(Thread.java:679)
--e89a8f502e0c64dcb904d1eef20a--