Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B27D199F8 for ; Fri, 8 Jun 2012 17:56:33 +0000 (UTC) Received: (qmail 53192 invoked by uid 500); 8 Jun 2012 17:56:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 53169 invoked by uid 500); 8 Jun 2012 17:56:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 53161 invoked by uid 99); 8 Jun 2012 17:56:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jun 2012 17:56:31 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of driftx@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-we0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jun 2012 17:56:26 +0000 Received: by werb13 with SMTP id b13so986725wer.31 for ; Fri, 08 Jun 2012 10:56:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=icPTRljHLGiklYv+DuQXoF9k1AkgCIMlYP+H2G4X0Hg=; b=vHRUmYkL+KtYT5ZrrUZN+dBEVfVR5eAUKpAyB3DAN49iRxk/V4iZJow9hCfSSJElGi SUqruJZ/ze73W4GWDG5xw5J3zKxevGI8JSrI0Ie9KN7Il5OUiY/TlLjdTFEoNbv/giZe reJ6pd6xt0I/LNq02tGHCoDGTHZeZV4teXuIbaW8Bft/SI3rnwskYNHwevcvUpN+LduX wIuGJotSCJyyA+pVj+16OdnVwRZ0AFZtD5VdeQ0dy9n0lf+O3JV21uH55g9T4gNfq6kK G4tmDW956jtj7xgWQg88IY88C8Kc2FuI64eZhVL6txzhFE1S/i7eBjdUGusu1GhhNmyY qmig== Received: by 10.216.205.5 with SMTP id i5mr1763735weo.6.1339178162301; Fri, 08 Jun 2012 10:56:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.47.67 with HTTP; Fri, 8 Jun 2012 10:55:42 -0700 (PDT) In-Reply-To: <054A19B0E41FDD43A58D2CC1E8C3FF5B0BDEE7FA@FISH-MBX-02.azaleos.net> References: <054A19B0E41FDD43A58D2CC1E8C3FF5B0BDEE7FA@FISH-MBX-02.azaleos.net> From: Brandon Williams Date: Fri, 8 Jun 2012 12:55:42 -0500 Message-ID: Subject: Re: Problem joining new node to cluster in 1.1.1 To: user@cassandra.apache.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org This sounds related to https://issues.apache.org/jira/browse/CASSANDRA-4251 On Thu, Jun 7, 2012 at 5:28 PM, Bryce Godfrey w= rote: > As the new node starts up I get this error before boostrap starts: > > > > INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0= /0 > serialized/live bytes, 1 ops) > > INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0 > serialized/live bytes, 1 ops) > > INFO 08:20:51,589 Completed flushing > /opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-Data= .db > (61 bytes) > > ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main] > > java.lang.IllegalArgumentException: value already present: 1015 > > =A0=A0=A0=A0=A0=A0=A0 at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:115= ) > > =A0=A0=A0=A0=A0=A0=A0 at > com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:= 111) > > =A0=A0=A0=A0=A0=A0=A0 at > com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96) > > =A0=A0=A0=A0=A0=A0=A0 at com.google.common.collect.HashBiMap.put(HashBiMa= p.java:84) > > =A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.config.Schema.load(Schema.j= ava:385) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:361) > > =A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.db.DefsTable.mergeSchema(De= fsTable.java:270) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(M= igrationManager.java:416) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.Executors$RunnableAdapter.c= all(Unknown > Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.FutureTask$Sync.innerRun(Un= known Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.FutureTask.run(Unknown Sour= ce) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.r= unTask(Unknown > Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.r= un(Unknown > Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Unknown Source) > > INFO 08:20:51,931 Enqueuing flush of > Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 op= s) > > INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178 > serialized/live bytes, 20 ops) > > > > > > Then it starts spewing these errors nonstop until I kill it. > > > > ERROR 08:21:45,959 Error in row mutation > > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find > cfId=3D1019 > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySe= rializer.java:126) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Row= Mutation.java:439) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Row= Mutation.java:447) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHand= ler.java:42) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :59) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.r= unTask(Unknown > Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.r= un(Unknown > Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Unknown Source) > > ERROR 08:21:45,814 Error in row mutation > > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find > cfId=3D1019 > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySe= rializer.java:126) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Row= Mutation.java:439) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Row= Mutation.java:447) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHand= ler.java:42) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :59) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.r= unTask(Unknown > Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.r= un(Unknown > Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Unknown Source) > > ERROR 08:21:45,813 Error in row mutation > > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find > cfId=3D1020 > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySe= rializer.java:126) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Row= Mutation.java:439) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(Row= Mutation.java:447) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHand= ler.java:42) > > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :59) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.r= unTask(Unknown > Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.r= un(Unknown > Source) > > =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Unknown Source) > > ERROR 08:21:45,813 Error in row mutation > > > > > > I=92m guessing the first error caused some column families to not be crea= ted? > But I=92m unsure on how to resolve this.