Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8FACE10D34 for ; Fri, 20 Dec 2013 12:08:23 +0000 (UTC) Received: (qmail 59115 invoked by uid 500); 20 Dec 2013 12:08:22 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 58845 invoked by uid 500); 20 Dec 2013 12:08:15 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 58802 invoked by uid 99); 20 Dec 2013 12:08:11 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Dec 2013 12:08:11 +0000 Date: Fri, 20 Dec 2013 12:08:11 +0000 (UTC) From: "Lajos Bodrogi (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-6224) CQL3 Column family / tables "disappear", get unconfigured columnfamily errors MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853891#comment-13853891 ] Lajos Bodrogi commented on CASSANDRA-6224: ------------------------------------------ Maybe i have a related problem. (Cassandra 2.0.3, Ubuntu 12.04, Cluster of 3 nodes) I got the following Perl error: ARGH! NEW PRIMARY FAILED: OPCODE_ERROR: unconfigured columnfamily source 8704 Protocol::CassandraCQL::Frame=SCALAR(0x10f79a0) And one node stopped responding (the java process was still running) while the others in the cluster kept working. Here are the server logs: INFO 12:23:26,299 flushing high-traffic column family CFS(Keyspace='vst8', ColumnFamily='series') (estimated 38461316 bytes) INFO 12:23:26,299 Enqueuing flush of Memtable-series@1926422961(11944355/38461316 serialized/live bytes, 230603 ops) INFO 12:23:26,300 Writing Memtable-series@1926422961(11944355/38461316 serialized/live bytes, 230603 ops) ERROR 12:23:26,511 Exception in thread Thread[FlushWriter:46,5,main] FSWriteError in /var/lib/cassandra/data/vst8/series/vst8-series-tmp-jb-760-Filter.db at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:462) at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:210) at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:288) at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:387) at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:320) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.write(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:282) at java.io.DataOutputStream.writeInt(DataOutputStream.java:197) at org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34) at org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44) at org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41) at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:455) ... 9 more ERROR 12:23:26,513 Stopping gossiper WARN 12:23:26,513 Stopping gossip by operator request INFO 12:23:26,513 Announcing shutdown ERROR 12:23:28,513 Stopping RPC server INFO 12:23:28,513 Stop listening to thrift clients ERROR 12:23:28,517 Stopping native transport INFO 12:23:28,532 Stop listening for CQL clients As you can see, the disk went full. > CQL3 Column family / tables "disappear", get unconfigured columnfamily errors > ----------------------------------------------------------------------------- > > Key: CASSANDRA-6224 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6224 > Project: Cassandra > Issue Type: Bug > Components: API, Core > Environment: Cassandra 2.0.1 Amazon AWS Ubuntu Single-node > Reporter: Constance Eustace > > We're seeing CQL3 tables seemingly arbitrarily disappear. Need to repair for prod meant we reconstructed the affected schema before nodetool repairs or similar attempts were done. > It seems to take a few days to appear. Volumes are not tremendously high yet... > Caused by: java.sql.SQLSyntaxErrorException: InvalidRequestException(why:unconfigured columnfamily entity_hierarchydef) > at org.apache.cassandra.cql.jdbc.CassandraPreparedStatement.(CassandraPreparedStatement.java:103) ~[cassandra-jdbc-1.2.5.jar:na] > at org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:388) ~[cassandra-jdbc-1.2.5.jar:na] > at org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:372) ~[cassandra-jdbc-1.2.5.jar:na] > at org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:50) ~[cassandra-jdbc-1.2.5.jar:na] > at org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281) ~[commons-dbcp-1.3.jar:1.3] > at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313) ~[commons-dbcp-1.3.jar:1.3] > at com.bestbuy.contentsystems.cupcake.storage.cassandra.cqlentity.CQL.tool.CassPSC.createPreparedStatement(CassPSC.java:61) ~[ingest-storage-QA-SNAPSHOT.jar:QA-SNAPSHOT] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:583) ~[spring-jdbc-3.2.4.RELEASE.jar:3.2.4.RELEASE] > ... 148 common frames omitted > Caused by: org.apache.cassandra.thrift.InvalidRequestException: null > at org.apache.cassandra.thrift.Cassandra$prepare_cql3_query_result.read(Cassandra.java:39567) ~[cassandra-thrift-1.2.8.jar:1.2.8] > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.7.0.jar:0.7.0] > at org.apache.cassandra.thrift.Cassandra$Client.recv_prepare_cql3_query(Cassandra.java:1625) ~[cassandra-thrift-1.2.8.jar:1.2.8] > at org.apache.cassandra.thrift.Cassandra$Client.prepare_cql3_query(Cassandra.java:1611) ~[cassandra-thrift-1.2.8.jar:1.2.8] > at org.apache.cassandra.cql.jdbc.CassandraConnection.prepare(CassandraConnection.java:517) ~[cassandra-jdbc-1.2.5.jar:na] > at org.apache.cassandra.cql.jdbc.CassandraConnection.prepare(CassandraConnection.java:532) ~[cassandra-jdbc-1.2.5.jar:na] > at org.apache.cassandra.cql.jdbc.CassandraPreparedStatement.(CassandraPreparedStatement.java:96) ~[cassandra-jdbc-1.2.5.jar:na] -- This message was sent by Atlassian JIRA (v6.1.4#6159)