cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Petrov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13113) test failure in auth_test.TestAuth.system_auth_ks_is_alterable_test
Date Thu, 23 Mar 2017 08:02:41 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937893#comment-15937893
] 

Alex Petrov commented on CASSANDRA-13113:
-----------------------------------------

I've investigated a bit deeper. Although in my opinion it's kind of a regression, even if
it's not super-serious, but it has some user-facing implications. I've ran {{bisect}} and
narrowed it down to [this commit|https://github.com/apache/cassandra/commit/c607d76413be81a0e125c5780e068d7ab7594612]


Checking logs reveals that before this commit, we had error messages in the form of:

{code}
Error from server: code=0100 [Bad credentials] message="Error during authentication of user
cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency
level QUORUM"
{code}

After, it's changed to 

{code}
Error from server: code=0000 [Server error] message="java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException:
Cannot achieve consistency level QUORUM"
{code}

I've checked underlying code and it looks like Guava was doing some unwrapping in case of
runtime exceptions on [cache loading|http://grepcode.com/file/repo1.maven.org/maven2/com.google.guava/guava/11.0/com/google/common/cache/LocalCache.java#2234]
(might be a wrong guava version but you get the idea). Previously, we had to unwrap the {{UncheckedExecutionException}}
in order to extract cause and [turn it into authentication exception|https://github.com/ifesdjeen/cassandra/commit/c607d76413be81a0e125c5780e068d7ab7594612#diff-ef1e335e8d51911f09bcc735b0632c5cL97],
in order to trigger a correct error code. Now, we don't have to since exception isn't un/rewrapped.


The stack trace of the other exception that was happening and causing {{Server error}} instead
of {{Bad Credentials}} was

{code}
        at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:487)
[main/:na]
        at org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:310)
[main/:na]
        at org.apache.cassandra.service.ClientState.login(ClientState.java:271) [main/:na]
        at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:80)
[main/:na]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
[main/:na]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
[main/:na]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
[main/:na]
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [main/:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
{code}

Consequently, I have removed guava-specific exception rewrapping. The other places (JMX permissions
cache, Credentials cache, Passwords cache and Permissions cache) look fine, with an exception
with Permission cache where we do re-wrap an exception but that doesn't change bubbling/error
code. 

|[trunk|https://github.com/apache/cassandra/compare/trunk...ifesdjeen:13367-trunk]|[dtest|https://cassci.datastax.com/job/ifesdjeen-13367-trunk-dtest/]|[testall|https://cassci.datastax.com/job/ifesdjeen-13367-trunk-testall/]|

I wanted to add that it might be not a very good style that we're using exceptions for a control
flow. We might want to think of another way to handle such things in future, at least for
the top-tier user facing return codes. Or at least as it was mentioned in [the comment|https://issues.apache.org/jira/browse/CASSANDRA-10855?focusedCommentId=15789267&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15789267],
we might want to cover such things with test for future (not necessarily unit tests even).

> test failure in auth_test.TestAuth.system_auth_ks_is_alterable_test
> -------------------------------------------------------------------
>
>                 Key: CASSANDRA-13113
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13113
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Testing
>            Reporter: Sean McCarthy
>            Assignee: Alex Petrov
>              Labels: dtest, test-failure
>         Attachments: node1_debug.log, node1_gc.log, node1.log, node2_debug.log, node2_gc.log,
node2.log, node3_debug.log, node3_gc.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1466/testReport/auth_test/TestAuth/system_auth_ks_is_alterable_test
> {code}
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 358, in run
>     self.tearDown()
>   File "/home/automaton/cassandra-dtest/dtest.py", line 582, in tearDown
>     raise AssertionError('Unexpected error in log, see stdout')
> {code}{code}
> Standard Output
> Unexpected error in node2 log, error: 
> ERROR [Native-Transport-Requests-1] 2017-01-08 21:10:55,056 Message.java:623 - Unexpected
exception during request; channel = [id: 0xf39c6dae, L:/127.0.0.2:9042 - R:/127.0.0.1:43640]
> java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot
achieve consistency level QUORUM
> 	at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503)
~[main/:na]
> 	at org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:310)
~[main/:na]
> 	at org.apache.cassandra.service.ClientState.login(ClientState.java:271) ~[main/:na]
> 	at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:80)
~[main/:na]
> 	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
[main/:na]
> 	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
[main/:na]
> 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
[main/:na]
> 	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [main/:na]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency
level QUORUM
> 	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334)
~[main/:na]
> 	at org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162)
~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1734)
~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1696) ~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1642) ~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1557) ~[main/:na]
> 	at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:964)
~[main/:na]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:282)
~[main/:na]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:252)
~[main/:na]
> 	at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:511)
~[main/:na]
> 	at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:493)
~[main/:na]
> 	... 13 common frames omitted
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message