cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Durity, Sean R" <SEAN_R_DUR...@homedepot.com>
Subject RE: [EXTERNAL] Re: Error after 3.1.0 to 3.11.2 upgrade
Date Mon, 14 May 2018 13:11:13 GMT
A couple additional things:


-          Make sure that you ran repair on the system_auth keyspace on all nodes after changing
the RF

-          If you are not often changing roles/permissions, you might look to increase permissions_validity_in_ms
and roles_validity_in_ms so they are not being fetched all the time (especially with the internal
Cassandra Authorizer/Authenticator).


Sean Durity

From: Jeff Jirsa <jjirsa@gmail.com>
Sent: Saturday, May 12, 2018 9:21 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Error after 3.1.0 to 3.11.2 upgrade

RF of one means all auth requests go to the same node, so they’re more likely to time out
if that host is overloaded or restarts

Increasing it distributed the queries among more hosts

--
Jeff Jirsa


On May 12, 2018, at 6:11 AM, Abdul Patel <abd786.ap@gmail.com<mailto:abd786.ap@gmail.com>>
wrote:
Yeah found that all had 3 replication factor and system_auth had 1 , chnaged to 3 now ..so
was this issue due to system_auth replication facyor mismatch?

On Saturday, May 12, 2018, Hannu Kröger <hkroger@gmail.com<mailto:hkroger@gmail.com>>
wrote:
Hi,

Did you check replication strategy and amounts of replicas of system_auth keyspace?

Hannu

Abdul Patel <abd786.ap@gmail.com<mailto:abd786.ap@gmail.com>> kirjoitti 12.5.2018
kello 5.21:
No applicatiom isnt impacted ..no complains ..
Also its an 4 node cluster in lower non production and all are on same version.

On Friday, May 11, 2018, Jeff Jirsa <jjirsa@gmail.com<mailto:jjirsa@gmail.com>>
wrote:
The read is timing out - is the cluster healthy? Is it fully upgraded or mixed versions? Repeated
isn’t great, but is the application impacted?
--
Jeff Jirsa


On May 12, 2018, at 6:17 AM, Abdul Patel <abd786.ap@gmail.com<mailto:abd786.ap@gmail.com>>
wrote:
Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this is repeatedly coming
, whats solution for this?

WARN  [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 CassandraAuthorizer.java:96 -
CassandraAuthorizer failed to authorize #<User cas_sar_dit> for <table et_session.session>
ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 ErrorMessage.java:384 - Unexpected
exception during request
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only
0 responses.
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
~[guava-18.0.jar:na]
        at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260)
~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221)
~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530)
~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507)
~[apache-cassandra-3.11.2.jar:3.11.2]

On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa <jjirsa@gmail.com<mailto:jjirsa@gmail.com>>
wrote:
That looks like Cassandra 3.10 not 3.11.2

It’s also just the auth cache failing to refresh - if it’s transient it’s probably not
a big deal. If it continues then there may be an issue with the cache refresher.
--
Jeff Jirsa


On May 12, 2018, at 5:55 AM, Abdul Patel <abd786.ap@gmail.com<mailto:abd786.ap@gmail.com>>
wrote:
HI All,

Seen below stack trace messages , in errorlog  one day after upgrade.
one of the blogs said this might be due to old drivers, but not sure on it.

FYI :

INFO  [HANDSHAKE-/10.152.205.150<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.152.205.150&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=Y8XD0Q963uyonwtvyXypGAKBaC4gihfqvxDECM8v8hU&s=5W-dlzrZjStoaJ_FJmmLmqNGfnZFIAgZ-_B8hMtHLyw&e=>]
2018-05-09 10:22:27,160 OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.152.205.150&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=Y8XD0Q963uyonwtvyXypGAKBaC4gihfqvxDECM8v8hU&s=5W-dlzrZjStoaJ_FJmmLmqNGfnZFIAgZ-_B8hMtHLyw&e=>
DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 10:22:27,160 OutboundTcpConnection.java:482
- Done connecting to /10.152.205.150<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.152.205.150&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=Y8XD0Q963uyonwtvyXypGAKBaC4gihfqvxDECM8v8hU&s=5W-dlzrZjStoaJ_FJmmLmqNGfnZFIAgZ-_B8hMtHLyw&e=>
ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 ErrorMessage.java:384 - Unexpected
exception during request
com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level LOCAL_ONE
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
~[guava-18.0.jar:na]
        at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) ~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) ~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
[apache-cassandra-3.10.jar:3.10]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
[apache-cassandra-3.10.jar:3.10]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException:
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
~[guava-18.0.jar:na]
        at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.PermissionsCache$$Lambda$269/275740217.apply(Unknown
Source) ~[na:na]
        at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10]
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
        ... 26 common frames omitted
Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException:
Cannot achieve consistency level LOCAL_ONE
        at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:513)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:481)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:285)
~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) ~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.auth.RolesCache$$Lambda$280/1091297529.apply(Unknown Source)
~[na:na]



________________________________

The information in this Internet Email is confidential and may be legally privileged. It is
intended solely for the addressee. Access to this Email by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject to the terms and
conditions expressed in any applicable governing The Home Depot terms of business or client
engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy
and content of this attachment and for any damages or losses arising from any inaccuracies,
errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature,
which may be contained in this attachment and shall not be liable for direct, indirect, consequential
or special damages in connection with this e-mail message or its attachment.
Mime
View raw message