cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdul Patel <abd786...@gmail.com>
Subject Re: Error after 3.1.0 to 3.11.2 upgrade
Date Sat, 12 May 2018 13:11:50 GMT
Yeah found that all had 3 replication factor and system_auth had 1 ,
chnaged to 3 now ..so was this issue due to system_auth replication facyor
mismatch?

On Saturday, May 12, 2018, Hannu Kröger <hkroger@gmail.com> wrote:

> Hi,
>
> Did you check replication strategy and amounts of replicas of system_auth
> keyspace?
>
> Hannu
>
> Abdul Patel <abd786.ap@gmail.com> kirjoitti 12.5.2018 kello 5.21:
>
> No applicatiom isnt impacted ..no complains ..
> Also its an 4 node cluster in lower non production and all are on same
> version.
>
> On Friday, May 11, 2018, Jeff Jirsa <jjirsa@gmail.com> wrote:
>
>> The read is timing out - is the cluster healthy? Is it fully upgraded or
>> mixed versions? Repeated isn’t great, but is the application impacted?
>>
>> --
>> Jeff Jirsa
>>
>>
>> On May 12, 2018, at 6:17 AM, Abdul Patel <abd786.ap@gmail.com> wrote:
>>
>> Seems its coming from 3.10, got bunch of them today for 3.11.2, so if
>> this is repeatedly coming , whats solution for this?
>>
>> WARN  [Native-Transport-Requests-24] 2018-05-11 16:46:20,938
>> CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize
>> #<User cas_sar_dit> for <table et_session.session>
>> ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940
>> ErrorMessage.java:384 - Unexpected exception during request
>> com.google.common.util.concurrent.UncheckedExecutionException:
>> java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException:
>> Operation timed out - received only 0 responses.
>>         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
>> ~[guava-18.0.jar:na]
>>         at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
>> ~[guava-18.0.jar:na]
>>         at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
>> ~[guava-18.0.jar:na]
>>         at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
>> ~[guava-18.0.jar:na]
>>         at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.service.ClientState.checkPermissionOnRe
>> sourceChain(ClientState.java:368) ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>         at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507)
>> ~[apache-cassandra-3.11.2.jar:3.11.2]
>>
>> On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa <jjirsa@gmail.com> wrote:
>>
>>> That looks like Cassandra 3.10 not 3.11.2
>>>
>>> It’s also just the auth cache failing to refresh - if it’s transient
>>> it’s probably not a big deal. If it continues then there may be an issue
>>> with the cache refresher.
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On May 12, 2018, at 5:55 AM, Abdul Patel <abd786.ap@gmail.com> wrote:
>>>
>>> HI All,
>>>
>>> Seen below stack trace messages , in errorlog  one day after upgrade.
>>> one of the blogs said this might be due to old drivers, but not sure on
>>> it.
>>>
>>> FYI :
>>>
>>> INFO  [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160
>>> OutboundTcpConnection.java:510 - Handshaking version with /
>>> 10.152.205.150
>>> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09
>>> 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to /
>>> 10.152.205.150
>>> ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971
>>> ErrorMessage.java:384 - Unexpected exception during request
>>> com.google.common.util.concurrent.UncheckedExecutionException:
>>> com.google.common.util.concurrent.UncheckedExecutionException:
>>> java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException:
>>> Cannot achieve consistency level LOCAL_ONE
>>>         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
>>> ~[guava-18.0.jar:na]
>>>         at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
>>> ~[guava-18.0.jar:na]
>>>         at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
>>> ~[guava-18.0.jar:na]
>>>         at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
>>> ~[guava-18.0.jar:na]
>>>         at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.service.ClientState.authorize(ClientState.java:419)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.service.ClientState.checkPermissionOnRe
>>> sourceChain(ClientState.java:352) ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
>>> [apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
>>> [apache-cassandra-3.10.jar:3.10]
>>>         at io.netty.channel.SimpleChannelInboundHandler.channelRead(Sim
>>> pleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.
>>> 0.39.Final]
>>>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannel
>>> Read(AbstractChannelHandlerContext.java:366)
>>> [netty-all-4.0.39.Final.jar:4.0.39.Final]
>>>         at io.netty.channel.AbstractChannelHandlerContext.access$600(Ab
>>> stractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.
>>> 0.39.Final]
>>>         at io.netty.channel.AbstractChannelHandlerContext$7.run(Abstrac
>>> tChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.
>>> 0.39.Final]
>>>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_51]
>>>         at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorSe
>>> rvice$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
>>> [apache-cassandra-3.10.jar:3.10]
>>>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
>>> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
>>> java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException:
>>> Cannot achieve consistency level LOCAL_ONE
>>>         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
>>> ~[guava-18.0.jar:na]
>>>         at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
>>> ~[guava-18.0.jar:na]
>>>         at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
>>> ~[guava-18.0.jar:na]
>>>         at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
>>> ~[guava-18.0.jar:na]
>>>         at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.PermissionsCache$$Lambda$269/275740217.apply(Unknown
>>> Source) ~[na:na]
>>>         at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
>>> ~[guava-18.0.jar:na]
>>>         at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
>>> ~[guava-18.0.jar:na]
>>>         at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
>>> ~[guava-18.0.jar:na]
>>>         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
>>> ~[guava-18.0.jar:na]
>>>         ... 26 common frames omitted
>>> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException:
>>> Cannot achieve consistency level LOCAL_ONE
>>>         at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:513)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:481)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:285)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>>         at org.apache.cassandra.auth.RolesCache$$Lambda$280/1091297529.apply(Unknown
>>> Source) ~[na:na]
>>>
>>>
>>

Mime
View raw message