Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E241311477 for ; Tue, 26 Aug 2014 14:25:01 +0000 (UTC) Received: (qmail 20658 invoked by uid 500); 26 Aug 2014 14:25:01 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 20607 invoked by uid 500); 26 Aug 2014 14:25:01 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 20353 invoked by uid 99); 26 Aug 2014 14:25:01 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Aug 2014 14:25:01 +0000 Date: Tue, 26 Aug 2014 14:25:01 +0000 (UTC) From: "Igor Zubchenok (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-7828) New node cannot be joined if a value in composite type column is dropped (description updated) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-7828?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Zubchenok updated CASSANDRA-7828: -------------------------------------- Summary: New node cannot be joined if a value in composite type column = is dropped (description updated) (was: New node cannot be joined if a valu= e in composite type column is dropped) > New node cannot be joined if a value in composite type column is dropped = (description updated) > -------------------------------------------------------------------------= --------------------- > > Key: CASSANDRA-7828 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7828 > Project: Cassandra > Issue Type: Bug > Components: Core > Reporter: Igor Zubchenok > > I get a *RuntimeException* at new node system.log on bootstrapping a new = DC: > {code:title=3Dsystem.out - RuntimeException caused by IllegalArgumentExce= ption in Buffer.limit|borderStyle=3Dsolid} > INFO [NonPeriodicTasks:1] 2014-08-26 15:43:01,030 SecondaryIndexManager.j= ava (line 137) Submitting index build of [myColumnFamily.myColumnFamily_myC= olumn] for data in SSTableReader(path=3D'/var/lib/cassandra/data/testbug/my= ColumnFamily/testbug-myColumnFamily-jb-1-Data.db') > ERROR [CompactionExecutor:2] 2014-08-26 15:43:01,035 CassandraDaemon.java= (line 199) Exception in thread Thread[CompactionExecutor:2,1,main] > java.lang.IllegalArgumentException > =09at java.nio.Buffer.limit(Buffer.java:267) > =09at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.= java:587) > =09at org.apache.cassandra.utils.ByteBufferUtil.readBytesWithShortLength(= ByteBufferUtil.java:596) > =09at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(Abstr= actCompositeType.java:61) > =09at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(Abstr= actCompositeType.java:36) > =09at org.apache.cassandra.dht.LocalToken.compareTo(LocalToken.java:44) > =09at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85= ) > =09at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:36= ) > =09at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(Concurre= ntSkipListMap.java:727) > =09at java.util.concurrent.ConcurrentSkipListMap.findNode(ConcurrentSkipL= istMap.java:789) > =09at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipList= Map.java:828) > =09at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMa= p.java:1626) > =09at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215) > =09at org.apache.cassandra.db.Memtable.put(Memtable.java:173) > =09at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.j= ava:900) > =09at org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex= .insert(AbstractSimplePerColumnSecondaryIndex.java:107) > =09at org.apache.cassandra.db.index.SecondaryIndexManager.indexRow(Second= aryIndexManager.java:441) > =09at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:413) > =09at org.apache.cassandra.db.index.SecondaryIndexBuilder.build(Secondary= IndexBuilder.java:62) > =09at org.apache.cassandra.db.compaction.CompactionManager$9.run(Compacti= onManager.java:834) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > ERROR [NonPeriodicTasks:1] 2014-08-26 15:43:01,035 CassandraDaemon.java (= line 199) Exception in thread Thread[NonPeriodicTasks:1,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: java= .lang.IllegalArgumentException > =09at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.jav= a:413) > =09at org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecon= daryIndexes(SecondaryIndexManager.java:142) > =09at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnab= le.run(StreamReceiveTask.java:113) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.access$201(ScheduledThreadPoolExecutor.java:178) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.run(ScheduledThreadPoolExecutor.java:292) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgu= mentException > =09at java.util.concurrent.FutureTask.report(FutureTask.java:122) > =09at java.util.concurrent.FutureTask.get(FutureTask.java:188) > =09at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.jav= a:409) > =09... 9 more > Caused by: java.lang.IllegalArgumentException > =09at java.nio.Buffer.limit(Buffer.java:267) > =09at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.= java:587) > =09at org.apache.cassandra.utils.ByteBufferUtil.readBytesWithShortLength(= ByteBufferUtil.java:596) > =09at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(Abstr= actCompositeType.java:61) > =09at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(Abstr= actCompositeType.java:36) > =09at org.apache.cassandra.dht.LocalToken.compareTo(LocalToken.java:44) > =09at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85= ) > =09at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:36= ) > =09at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(Concurre= ntSkipListMap.java:727) > =09at java.util.concurrent.ConcurrentSkipListMap.findNode(ConcurrentSkipL= istMap.java:789) > =09at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipList= Map.java:828) > =09at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMa= p.java:1626) > =09at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215) > =09at org.apache.cassandra.db.Memtable.put(Memtable.java:173) > =09at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.j= ava:900) > =09at org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex= .insert(AbstractSimplePerColumnSecondaryIndex.java:107) > =09at org.apache.cassandra.db.index.SecondaryIndexManager.indexRow(Second= aryIndexManager.java:441) > =09at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:413) > =09at org.apache.cassandra.db.index.SecondaryIndexBuilder.build(Secondary= IndexBuilder.java:62) > =09at org.apache.cassandra.db.compaction.CompactionManager$9.run(Compacti= onManager.java:834) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09... 3 more > ERROR [CompactionExecutor:2] 2014-08-26 15:43:01,036 CassandraDaemon.java= (line 199) Exception in thread Thread[CompactionExecutor:2,1,main] > java.lang.IllegalArgumentException > =09at java.nio.Buffer.limit(Buffer.java:267) > =09at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.= java:587) > =09at org.apache.cassandra.utils.ByteBufferUtil.readBytesWithShortLength(= ByteBufferUtil.java:596) > =09at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(Abstr= actCompositeType.java:61) > =09at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(Abstr= actCompositeType.java:36) > =09at org.apache.cassandra.dht.LocalToken.compareTo(LocalToken.java:44) > =09at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85= ) > =09at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:36= ) > =09at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(Concurre= ntSkipListMap.java:727) > =09at java.util.concurrent.ConcurrentSkipListMap.findNode(ConcurrentSkipL= istMap.java:789) > =09at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipList= Map.java:828) > =09at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMa= p.java:1626) > =09at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215) > =09at org.apache.cassandra.db.Memtable.put(Memtable.java:173) > =09at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.j= ava:900) > =09at org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex= .insert(AbstractSimplePerColumnSecondaryIndex.java:107) > =09at org.apache.cassandra.db.index.SecondaryIndexManager.indexRow(Second= aryIndexManager.java:441) > =09at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:413) > =09at org.apache.cassandra.db.index.SecondaryIndexBuilder.build(Secondary= IndexBuilder.java:62) > =09at org.apache.cassandra.db.compaction.CompactionManager$9.run(Compacti= onManager.java:834) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > {code} > My steps to reproduce (=D0=A1assandra 2.0.9): > # In one node datacenter DC1 create a keyspace with NetworkTopologyStrate= gy, options DC1=3D1, DC2=3D1. > # Create a CF with a secondary index on a CompositeType(Int32Type, UUIDTy= pe). > # Write any composite value to a row. > # Drop the value. > # Write another composite value to another row. > # Drop second value. > # Bootstrap add a new node in DC2. > # DC2 gets an exception. > The similar issue I got on indexed composite column in a single 3 nodes = DC with SimpleStrategy and replication_factor=3D3 (cassandra 1.2.18). > I use Astyanax 2.0.1 (I tested on Astyanax 1.56.49 as well), here is the = code that generates keyspace (steps 1-6) that fails on a new datacenter nod= e bootstrap: > {code:title=3DCassandraBugTest.java|borderStyle=3Dsolid} > package test; > import com.netflix.astyanax.AstyanaxContext; > import com.netflix.astyanax.Cluster; > import com.netflix.astyanax.Keyspace; > import com.netflix.astyanax.connectionpool.NodeDiscoveryType; > import com.netflix.astyanax.connectionpool.exceptions.BadRequestException= ; > import com.netflix.astyanax.connectionpool.exceptions.ConnectionException= ; > import com.netflix.astyanax.connectionpool.impl.ConnectionPoolConfigurati= onImpl; > import com.netflix.astyanax.connectionpool.impl.ConnectionPoolType; > import com.netflix.astyanax.connectionpool.impl.Slf4jConnectionPoolMonito= rImpl; > import com.netflix.astyanax.ddl.ColumnDefinition; > import com.netflix.astyanax.ddl.ColumnFamilyDefinition; > import com.netflix.astyanax.ddl.KeyspaceDefinition; > import com.netflix.astyanax.impl.AstyanaxConfigurationImpl; > import com.netflix.astyanax.model.ColumnFamily; > import com.netflix.astyanax.model.Composite; > import com.netflix.astyanax.model.ConsistencyLevel; > import com.netflix.astyanax.serializers.*; > import com.netflix.astyanax.thrift.ThriftFamilyFactory; > import org.junit.Test; > import org.junit.runner.RunWith; > import org.junit.runners.JUnit4; > import java.util.HashMap; > import java.util.Map; > import java.util.UUID; > @RunWith(JUnit4.class) > public class CassandraBugTest > { > public static final String COLUMN_NAME =3D "myColumn"; > public static final String COLUMN_FAMILY_NAME =3D "myColumnFamily"; > private static final String KEYSPACE_NAME =3D "testbug7"; > private static final String CLUSTER_NAME =3D "Test Cluster"; > private static final String SEEDS =3D "dc1n1"; > @Test > public void test() throws ConnectionException, InterruptedException > { > AstyanaxContext.Builder builder =3D new AstyanaxContext.Builder() > .forCluster(CLUSTER_NAME) > .forKeyspace(KEYSPACE_NAME) > .withAstyanaxConfiguration(new AstyanaxConfigurationImpl(= ) > .setDiscoveryType(NodeDiscoveryType.RING_= DESCRIBE) > .setConnectionPoolType(ConnectionPoolType= .TOKEN_AWARE) > .setCqlVersion("3.1.7") > .setTargetCassandraVersion("2.0.9") > .setDefaultWriteConsistencyLevel(Consiste= ncyLevel.CL_LOCAL_QUORUM) > .setDefaultReadConsistencyLevel(Consisten= cyLevel.CL_LOCAL_QUORUM) > ) > .withConnectionPoolConfiguration(new ConnectionPoolConfig= urationImpl("TaxiClusterConnectionPool") > .setInitConnsPerHost(1) > .setMaxConnsPerHost(8) > .setSeeds(SEEDS) > ) > .withConnectionPoolMonitor(new Slf4jConnectionPoolMonitor= Impl()); > AstyanaxContext clusterContext =3D builder.buildCluster(= ThriftFamilyFactory.getInstance()); > clusterContext.start(); > Cluster cluster =3D clusterContext.getClient(); > Map options =3D new HashMap<>(); > options.put("DC1", "1"); > options.put("DC2", "1"); > KeyspaceDefinition keyspaceDef =3D cluster.makeKeyspaceDefinition= () > .setName(KEYSPACE_NAME) > .setStrategyClass("Networ= kTopologyStrategy") > .setStrategyOptions(optio= ns); > try > { > cluster.dropKeyspace(KEYSPACE_NAME); > } > catch (BadRequestException e) > { > } > cluster.addKeyspace(keyspaceDef); > ColumnFamily profiles =3D new ColumnFamily<>(COLUMN= _FAMILY_NAME, UUIDSerializer.get(), StringSerializer.get()); > ColumnFamilyDefinition columnFamilyDef =3D cluster.makeColumnFami= lyDefinition() > .setName(profiles= .getName()) > .setKeyspace(KEYS= PACE_NAME) > .setKeyValidation= Class(profiles.getKeySerializer().getComparatorType().getTypeName()) > .setComparatorTyp= e(profiles.getColumnSerializer().getComparatorType().getTypeName()); > ColumnDefinition columnDef =3D cluster.makeColumnDefinition() > .setValidationClass( > ComparatorType.COMPOS= ITETYPE.getTypeName() + > "(" + ComparatorType.= INT32TYPE.getTypeName() + > "," + ComparatorType.= UUIDTYPE.getTypeName() + > ")" > ) > // Use BYTESTYPE here for val= idation class for workaround, then no bug occurs. > // .setValidationClass(Compara= torType.BYTESTYPE.getTypeName()) > .setName(COLUMN_NAME) > .setIndex(COLUMN_FAMILY_NAME = + "_" + COLUMN_NAME, "KEYS"); > columnFamilyDef.addColumnDefinition(columnDef); > cluster.addColumnFamily(columnFamilyDef); > AstyanaxContext keyspaceContext =3D builder.buildKeyspa= ce(ThriftFamilyFactory.getInstance()); > keyspaceContext.start(); > Keyspace keyspace =3D keyspaceContext.getClient(); > for (int i =3D 0; i < 100; ++i) > { > UUID id =3D UUID.randomUUID(); > Composite value =3D new Composite(); > value.addComponent(1, IntegerSerializer.get()); > value.addComponent(UUID.randomUUID(), UUIDSerializer.get()); > value.setSerializersByPosition(IntegerSerializer.get(), UUIDS= erializer.get()); > keyspace.prepareColumnMutation(profiles, id, COLUMN_NAME) > .putValue(value, CompositeSerializer.get(), null) > .execute(); > keyspace.prepareColumnMutation(profiles, id, COLUMN_NAME) > .deleteColumn() > .execute(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)