Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 16952188C1 for ; Fri, 4 Dec 2015 20:33:12 +0000 (UTC) Received: (qmail 87900 invoked by uid 500); 4 Dec 2015 20:33:11 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 87873 invoked by uid 500); 4 Dec 2015 20:33:11 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 87859 invoked by uid 99); 4 Dec 2015 20:33:11 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Dec 2015 20:33:11 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id BB3922C1F61 for ; Fri, 4 Dec 2015 20:33:11 +0000 (UTC) Date: Fri, 4 Dec 2015 20:33:11 +0000 (UTC) From: "Sam Tunnicliffe (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-8194) Reading from Auth table should not be in the request path MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-8194: --------------------------------------- Component/s: Distributed Metadata > Reading from Auth table should not be in the request path > --------------------------------------------------------- > > Key: CASSANDRA-8194 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 > Project: Cassandra > Issue Type: Improvement > Components: Distributed Metadata > Reporter: Vishy Kasar > Assignee: Sam Tunnicliffe > Priority: Minor > Fix For: 2.0.12, 2.1.3 > > Attachments: 8194-V2.patch, 8194-V3.txt, 8194-V4.txt, 8194-V5.txt, 8194.patch, CacheTest2.java > > > We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. > We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. > We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. > com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) > at com.google.common.cache.LocalCache.get(LocalCache.java:3990) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) > at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) > at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) > at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) > at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) > at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) > at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) > at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) > at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) > at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) > at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) > at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) > at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) > at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) > at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) > at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) > at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) > at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) > ... 19 more > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) > at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) > at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) > ... 28 more > ERROR [Thrift:17232] 2014-10-24 05:06:51,004 CustomTThreadPoolServer.java (line 224) Error occurred during processing of message. > com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) > at com.google.common.cache.LocalCache.get(LocalCache.java:3990) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) > at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) > at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) > at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) > at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) > at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) > at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:116) > at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) > at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) > at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) > at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) > at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) > at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) > at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) > at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) > at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) > at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) > at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) > ... 19 more > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) > at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) > at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) > ... 28 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)