cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lajos Bodrogi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6224) CQL3 Column family / tables "disappear", get unconfigured columnfamily errors
Date Fri, 20 Dec 2013 12:08:11 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853891#comment-13853891
] 

Lajos Bodrogi commented on CASSANDRA-6224:
------------------------------------------

Maybe i have a related problem. (Cassandra 2.0.3, Ubuntu 12.04, Cluster of 3 nodes)

I got the following Perl error: 
    ARGH! NEW PRIMARY FAILED: OPCODE_ERROR: unconfigured columnfamily source 8704 Protocol::CassandraCQL::Frame=SCALAR(0x10f79a0)
And one node stopped responding (the java process was still running) while the others in the
cluster kept working.

Here are the server logs:
 INFO 12:23:26,299 flushing high-traffic column family CFS(Keyspace='vst8', ColumnFamily='series')
(estimated 38461316 bytes)
 INFO 12:23:26,299 Enqueuing flush of Memtable-series@1926422961(11944355/38461316 serialized/live
bytes, 230603 ops)
 INFO 12:23:26,300 Writing Memtable-series@1926422961(11944355/38461316 serialized/live bytes,
230603 ops)
ERROR 12:23:26,511 Exception in thread Thread[FlushWriter:46,5,main]
FSWriteError in /var/lib/cassandra/data/vst8/series/vst8-series-tmp-jb-760-Filter.db
	at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:462)
	at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:210)
	at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:288)
	at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:387)
	at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:320)
	at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.write(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:282)
	at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
	at org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)
	at org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
	at org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)
	at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:455)
	... 9 more
ERROR 12:23:26,513 Stopping gossiper
 WARN 12:23:26,513 Stopping gossip by operator request
 INFO 12:23:26,513 Announcing shutdown
ERROR 12:23:28,513 Stopping RPC server
 INFO 12:23:28,513 Stop listening to thrift clients
ERROR 12:23:28,517 Stopping native transport
 INFO 12:23:28,532 Stop listening for CQL clients

As you can see, the disk went full. 

> CQL3 Column family / tables "disappear", get unconfigured columnfamily errors
> -----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-6224
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6224
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API, Core
>         Environment: Cassandra 2.0.1 Amazon AWS Ubuntu Single-node
>            Reporter: Constance Eustace
>
> We're seeing CQL3 tables seemingly arbitrarily disappear. Need to repair for prod meant
we reconstructed the affected schema before nodetool repairs or similar attempts were done.

> It seems to take a few days to appear. Volumes are not tremendously high yet...
> Caused by: java.sql.SQLSyntaxErrorException: InvalidRequestException(why:unconfigured
columnfamily entity_hierarchydef)
> 	at org.apache.cassandra.cql.jdbc.CassandraPreparedStatement.<init>(CassandraPreparedStatement.java:103)
~[cassandra-jdbc-1.2.5.jar:na]
> 	at org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:388)
~[cassandra-jdbc-1.2.5.jar:na]
> 	at org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:372)
~[cassandra-jdbc-1.2.5.jar:na]
> 	at org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:50)
~[cassandra-jdbc-1.2.5.jar:na]
> 	at org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
~[commons-dbcp-1.3.jar:1.3]
> 	at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
~[commons-dbcp-1.3.jar:1.3]
> 	at com.bestbuy.contentsystems.cupcake.storage.cassandra.cqlentity.CQL.tool.CassPSC.createPreparedStatement(CassPSC.java:61)
~[ingest-storage-QA-SNAPSHOT.jar:QA-SNAPSHOT]
> 	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:583) ~[spring-jdbc-3.2.4.RELEASE.jar:3.2.4.RELEASE]
> 	... 148 common frames omitted
> Caused by: org.apache.cassandra.thrift.InvalidRequestException: null
> 	at org.apache.cassandra.thrift.Cassandra$prepare_cql3_query_result.read(Cassandra.java:39567)
~[cassandra-thrift-1.2.8.jar:1.2.8]
> 	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.7.0.jar:0.7.0]
> 	at org.apache.cassandra.thrift.Cassandra$Client.recv_prepare_cql3_query(Cassandra.java:1625)
~[cassandra-thrift-1.2.8.jar:1.2.8]
> 	at org.apache.cassandra.thrift.Cassandra$Client.prepare_cql3_query(Cassandra.java:1611)
~[cassandra-thrift-1.2.8.jar:1.2.8]
> 	at org.apache.cassandra.cql.jdbc.CassandraConnection.prepare(CassandraConnection.java:517)
~[cassandra-jdbc-1.2.5.jar:na]
> 	at org.apache.cassandra.cql.jdbc.CassandraConnection.prepare(CassandraConnection.java:532)
~[cassandra-jdbc-1.2.5.jar:na]
> 	at org.apache.cassandra.cql.jdbc.CassandraPreparedStatement.<init>(CassandraPreparedStatement.java:96)
~[cassandra-jdbc-1.2.5.jar:na]



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message