cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michal Lefler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12575) A Cassandra node gets corrupted and contains only "key" and "column1" columns
Date Thu, 01 Sep 2016 06:25:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454475#comment-15454475
] 

Michal Lefler commented on CASSANDRA-12575:
-------------------------------------------

Sorry, but we resynched the problematic node already, and I have no idea how to reproduce.
However, restarting the node did not help, and so does a full repair, that was completed successfully
BTW, but did not help... 

> A Cassandra node gets corrupted and contains only "key" and "column1" columns
> -----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12575
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12575
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Cassandra version: cassandra-2.2.7-6
> We use a 3 data center environment, each data center contains two Cassandra nodes.
>            Reporter: Michal Lefler
>         Attachments: 122310b.jpg, Site2_Tables.JPG, system.log, system.log.1.zip
>
>
> We executed our application for a few hours, and all the nodes were functioning correctly.

> Then, after a few hours, we suddenly started getting the following exceptions:
> {code}
> com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name siteId in where
clause ('siteId = 'Site2-sdclab008-11-12'') [main_24] [TopologyManager.<init>()] com.datastax.driver.core.exceptions.InvalidQueryException:
Undefined name siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') 
>         at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)

>         at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)

>         at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)

>         at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)

>         at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)

>         at com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)

> {code}
> After some investigation, we found out that we got this kind of an exception whenever
we tried to query EACH ONE OF THE TABLES of one of our Cassandra nodes.
> When we tried to query the tables directly using CQL, we got the following result:
> All the tables only had one column named "key" for the partition key, and a "columnx"
column for each one of the clustering keys (column1, column2 etc.)
> For example:
> This query was executed on a healthy node: (siteId is the partition key, and flowType
is the clustering key)
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> siteId          | flowType  | currentVersion | flowName          | flowOwner     | flowState
| lockUpdatedTime  | previousVersion | rev
> -----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
> EMS-roger-01-02 | statusApi |        5.1-787 |      installation |               |  
   idle | 1472618536062180 |         5.1-787 |   1
> EMS-roger-01-02 |  topology |           null | topology_inserted | 10.240.32.130 |  
   idle | 1472618335876404 |            null |   1
> {code}
> The same query was executed on the problematic node:
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> key             | column1  
> -----------------+-----------
> EMS-roger-01-02 | statusApi 
> EMS-roger-01-02 |  topology
> {code}
> We tried to restart the node, and also to performed a full repair, but nothing helped.
> Finally we had to delete the node's data entirely, and resync it from the other nodes.
> We saw these lines on the Cassandra log (may or may not be related to the problem):
> {code}
> ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception encountered
during startup 
> java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured
table roles 
>         at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
[apache-cassandra-2.2.4.jar:2.2.4] 
> Caused by: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table
roles 
>         at org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         ... 8 common frames omitted 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message