cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sam Tunnicliffe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-12575) A Cassandra node gets corrupted and contains only "key" and "column1" columns
Date Wed, 31 Aug 2016 09:58:21 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-12575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sam Tunnicliffe updated CASSANDRA-12575:
----------------------------------------
    Description: 
We executed our application for a few hours, and all the nodes were functioning correctly.

Then, after a few hours, we suddenly started getting the following exceptions:
{code}
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name siteId in where
clause ('siteId = 'Site2-sdclab008-11-12'') [main_24] [TopologyManager.<init>()] com.datastax.driver.core.exceptions.InvalidQueryException:
Undefined name siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') 
        at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)

        at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)

        at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)

        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64) 
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39) 
        at com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)

{code}
After some investigation, we found out that we got this kind of an exception whenever we tried
to query EACH ONE OF THE TABLES of one of our Cassandra nodes.

When we tried to query the tables directly using CQL, we got the following result:
All the tables only had one column named "key" for the partition key, and a "columnx" column
for each one of the clustering keys (column1, column2 etc.)

For example:
This query was executed on a healthy node: (siteId is the partition key, and flowType is the
clustering key)
{code}
cqlsh> SELECT * from statusflow.flow ;

siteId          | flowType  | currentVersion | flowName          | flowOwner     | flowState
| lockUpdatedTime  | previousVersion | rev
-----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
EMS-roger-01-02 | statusApi |        5.1-787 |      installation |               |      idle
| 1472618536062180 |         5.1-787 |   1
EMS-roger-01-02 |  topology |           null | topology_inserted | 10.240.32.130 |      idle
| 1472618335876404 |            null |   1
{code}
The same query was executed on the problematic node:
{code}
cqlsh> SELECT * from statusflow.flow ;

key             | column1  
-----------------+-----------
EMS-roger-01-02 | statusApi 
EMS-roger-01-02 |  topology
{code}
We tried to restart the node, and also to performed a full repair, but nothing helped.
Finally we had to delete the node's data entirely, and resync it from the other nodes.

We saw these lines on the Cassandra log (may or may not be related to the problem):
{code}
ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception encountered during
startup 
java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured
table roles 
        at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315) [apache-cassandra-2.2.4.jar:2.2.4]

        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638) [apache-cassandra-2.2.4.jar:2.2.4]

Caused by: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table roles

        at org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        ... 8 common frames omitted 
{code}

  was:
We executed our application for a few hours, and all the nodes were functioning correctly.

Then, after a few hours, we suddenly started getting the following exceptions:
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name siteId in where
clause ('siteId = 'Site2-sdclab008-11-12'') [main_24] [TopologyManager.<init>()] com.datastax.driver.core.exceptions.InvalidQueryException:
Undefined name siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') 
        at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)

        at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)

        at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)

        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64) 
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39) 
        at com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)


After some investigation, we found out that we got this kind of an exception whenever we tried
to query EACH ONE OF THE TABLES of one of our Cassandra nodes.

When we tried to query the tables directly using CQL, we got the following result:
All the tables only had one column named "key" for the partition key, and a "columnx" column
for each one of the clustering keys (column1, column2 etc.)

For example:
This query was executed on a healthy node: (siteId is the partition key, and flowType is the
clustering key)
cqlsh> SELECT * from statusflow.flow ;

siteId          | flowType  | currentVersion | flowName          | flowOwner     | flowState
| lockUpdatedTime  | previousVersion | rev
-----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
EMS-roger-01-02 | statusApi |        5.1-787 |      installation |               |      idle
| 1472618536062180 |         5.1-787 |   1
EMS-roger-01-02 |  topology |           null | topology_inserted | 10.240.32.130 |      idle
| 1472618335876404 |            null |   1

The same query was executed on the problematic node:
cqlsh> SELECT * from statusflow.flow ;

key             | column1  
-----------------+-----------
EMS-roger-01-02 | statusApi 
EMS-roger-01-02 |  topology

We tried to restart the node, and also to performed a full repair, but nothing helped.
Finally we had to delete the node's data entirely, and resync it from the other nodes.

We saw these lines on the Cassandra log (may or may not be related to the problem):
ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception encountered during
startup 
java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured
table roles 
        at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315) [apache-cassandra-2.2.4.jar:2.2.4]

        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638) [apache-cassandra-2.2.4.jar:2.2.4]

Caused by: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table roles

        at org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
~[apache-cassandra-2.2.4.jar:2.2.4] 
        ... 8 common frames omitted 



> A Cassandra node gets corrupted and contains only "key" and "column1" columns
> -----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12575
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12575
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: We use a 3 data center environment, each data center contains two
Cassandra nodes.
>            Reporter: Michal Lefler
>         Attachments: 122310b.jpg, Site2_Tables.JPG, system.log, system.log.1.zip
>
>
> We executed our application for a few hours, and all the nodes were functioning correctly.

> Then, after a few hours, we suddenly started getting the following exceptions:
> {code}
> com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name siteId in where
clause ('siteId = 'Site2-sdclab008-11-12'') [main_24] [TopologyManager.<init>()] com.datastax.driver.core.exceptions.InvalidQueryException:
Undefined name siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') 
>         at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)

>         at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)

>         at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)

>         at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)

>         at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)

>         at com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)

> {code}
> After some investigation, we found out that we got this kind of an exception whenever
we tried to query EACH ONE OF THE TABLES of one of our Cassandra nodes.
> When we tried to query the tables directly using CQL, we got the following result:
> All the tables only had one column named "key" for the partition key, and a "columnx"
column for each one of the clustering keys (column1, column2 etc.)
> For example:
> This query was executed on a healthy node: (siteId is the partition key, and flowType
is the clustering key)
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> siteId          | flowType  | currentVersion | flowName          | flowOwner     | flowState
| lockUpdatedTime  | previousVersion | rev
> -----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
> EMS-roger-01-02 | statusApi |        5.1-787 |      installation |               |  
   idle | 1472618536062180 |         5.1-787 |   1
> EMS-roger-01-02 |  topology |           null | topology_inserted | 10.240.32.130 |  
   idle | 1472618335876404 |            null |   1
> {code}
> The same query was executed on the problematic node:
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> key             | column1  
> -----------------+-----------
> EMS-roger-01-02 | statusApi 
> EMS-roger-01-02 |  topology
> {code}
> We tried to restart the node, and also to performed a full repair, but nothing helped.
> Finally we had to delete the node's data entirely, and resync it from the other nodes.
> We saw these lines on the Cassandra log (may or may not be related to the problem):
> {code}
> ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception encountered
during startup 
> java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured
table roles 
>         at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
[apache-cassandra-2.2.4.jar:2.2.4] 
> Caused by: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table
roles 
>         at org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         at org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
~[apache-cassandra-2.2.4.jar:2.2.4] 
>         ... 8 common frames omitted 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message