Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	thelastpickle.com; b=y/T4FzL69zPvbmHUBH67tLPBM7hkYtlHbKivxHpLFl+
	NTPBHHNSsY4CwbLyaLqChfSlkYBqMe1KrcoOBHyk0a4P1kozH7gmdU9kV1gFlC5x
	72I2itV5/E9arBagQIZVwD11/y3QNxI7ULi2Sc+bzPQK4TRBkIxIi+bGhySTaNAI
	=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082.1)
Subject: Re: Attempt to assign id to existing column family.
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: <4D93BB2B.8020105@nicira.com>
Date: Fri, 1 Apr 2011 10:12:51 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <0CBA168A-BB12-4CBB-AF89-194188D15907@thelastpickle.com>
References: <4D93A696.1090408@nicira.com> <4D93BB2B.8020105@nicira.com>
To: user@cassandra.apache.org

There is no reason to change the RF on the system keyspace, it should =
probably not be allowed.

The system keyspace uses a LocalPartitioner and it's data is not =
replicated through the same mechanism as a user keyspace.=20
=20
Aaron
=20
On 31 Mar 2011, at 10:22, Jeremy Stribling wrote:

> On 03/30/2011 02:54 PM, Jeremy Stribling wrote:
>> After restarting a Cassandra 0.7.2 node, the node catches an =
exception during initialization and refuses to start:
>>=20
>> Caused by: org.apache.cassandra.config.ConfigurationException: =
Attempt to assign id to existing column family.
>>        at =
org.apache.cassandra.config.CFMetaData.map(CFMetaData.java:222)
>>        at =
org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescrip=
tor.java:477)=20
>>        ... 2 more
>>=20
>> Unlike a previous thread about this topic =
(http://www.mail-archive.com/user@cassandra.apache.org/msg09024.html), =
we are not trying to preserve the JVM across restarts.  The restart =
comes up in an entirely fresh JVM.  We are, however, embedding Cassandra =
in our application, but we're using the same steps used by =
AbstractCassandraDaemon to bring it up.
>>=20
>> Looking briefly through the code, the only way I see that this can =
happen is if loadSchemas tries to load information about the system =
table from storage (because the system table can be created in =
CFMetaData from the earlier =
DatabaseDescriptor.getTableMetaData(Table.SYSTEM_TABLE).values() call).  =
Or I guess the data on disk could have multiple entries under the same =
key, but the system table issue seems more likely to me.  Unfortunately =
the logging is not specific enough for me to tell which key it is =
failing with, and I haven't been able to reproduce this yet.
>>=20
>> One relevant piece of information might be that, before the restart, =
our application changed the replication factor of all the tables, =
including the system table:
>>=20
>> 2011-03-29 23:09:39,194 291146 [MigrationStage:1] INFO =
org.apache.cassandra.db.migration.Migration  - Applying migration =
9f371026-5a59-11e0-b23f-65ed1eced995 Update keyspace systemrep =
factor:1rep  strategy:LocalStrategy{...} to systemrep factor:3rep =
strategy:LocalStrategy{...}
>>=20
>> We're doing this in order to dynamically change the replication =
factor as new nodes are being added to the cluster (e.g., it starts off =
with one node and a repfactor of 1, and once there are three nodes, it =
increases the repfactor on all tables to 3).  Is it possible that =
migrations over the system table get written to disk in a way that would =
cause loadSchemas() during a restart to hit this exception?  Are we even =
allowed to change the replication factor of the system table?
>>=20
>=20
> I've confirmed that this happens when loading column family =
"IndexInfo" from the table "system" during the loadSchemas() call.  Does =
anyone know if there's a way to get around this?  Perhaps, like I =
theorized, it's not legit to change the replication factor on the system =
table.
>=20
>=20