From user-return-63029-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Fri Jan 18 15:20:36 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id AF3BC180647 for ; Fri, 18 Jan 2019 15:20:33 +0100 (CET) Received: (qmail 33247 invoked by uid 500); 18 Jan 2019 14:20:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 33237 invoked by uid 99); 18 Jan 2019 14:20:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jan 2019 14:20:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E52571807E5 for ; Fri, 18 Jan 2019 14:20:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.048 X-Spam-Level: ** X-Spam-Status: No, score=2.048 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 85ein9JNRQ_J for ; Fri, 18 Jan 2019 14:20:23 +0000 (UTC) Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id CA72860D90 for ; Fri, 18 Jan 2019 14:20:22 +0000 (UTC) Received: by mail-io1-f42.google.com with SMTP id x6so10779519ioa.9 for ; Fri, 18 Jan 2019 06:20:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=3YWzaPm4ZURAiYg3wUqtnr9l/Kb+xmcG19xZ0NfaCr0=; b=hYq9uPh/uCUYupg+tWGThpOdFz6oAB80uaECWblz4PieMpXVEta/+fCNdgbIYnoQCP 00F7aVYyU3WAfQJER694SDqorMyble2SZunJn/TwBE9z76/evEf2+Shpkcz/x1su4n4q 6c4jyNH8wPWDf4tE8Uwf7rHzP6JRFd7jtOqgoZWMfbHLjkzhYDi2NbiM5YfzVn75gnQK A5sSF+tP9MH+/xRkQALPKZqITeMmVWjHKs09F5enhMZgl6DlBbwTRzPHyXmVVQQDxDDX ccUfrW/8kg6DYrNJHBHezv2Hrh0Es8NN3YX0c1fLI71wTDJNBW6Fx6mxL6qNL3rpJ71p BJIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=3YWzaPm4ZURAiYg3wUqtnr9l/Kb+xmcG19xZ0NfaCr0=; b=FbC76xtQbQLkemXsxS5J+1hPAvhAsD650qO5k1/CvsDWdOQaDfj66KJW8AZ5gCJMCe 3cvLvTE0BsvbFih/8TjHPl/OjLfyjjYElmnXUOiXCRJIlA4LtiY83954GpMclV4UGbVC gcz3XoKvk+A81VulZKgIVogvCwstZ2Ad3LF37tKF6IIMMtHrzq1sUON7FYVlQ84eqQmG 6/luMM1WFI3jGkVL1lgRNuwFixRnZ82uPVxUgBzNh+CtXwLq5LA3O5onl6JB8W7ux4SK fNR+CjOzJ127z9yLgky04Yen5TIlyG0PJh5Uw7OciPoeUdNnb8V0Q5z7uQJAywFM+B2b q3qw== X-Gm-Message-State: AJcUuke21vF90ybKpViml4ANsxkfvvmdVYrm4y6PmtuwZkKHIVrB27mF 6lXXzz0KpYMvGWddGl+Ozb0bZ93fUEAv2J+WSRp+oQ== X-Google-Smtp-Source: ALg8bN7Ubj0r6owtz820uCKj8Msw/JAkGl+sCbyI3kRzV2oKF4VfzH8FmJ+Ad0e64tk4BdTqkxvLzZr1mgjV4dAyhgg= X-Received: by 2002:a5d:80d1:: with SMTP id h17mr10393659ior.158.1547821215614; Fri, 18 Jan 2019 06:20:15 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: rabii lamriq Date: Fri, 18 Jan 2019 15:20:04 +0100 Message-ID: Subject: Unexpected error during query + Operation timed out To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="000000000000bb8b6d057fbc350c" --000000000000bb8b6d057fbc350c Content-Type: text/plain; charset="UTF-8" Hi, We have some Timeout frequently in our node cassandra, can any one help me to analyse what can be the cause. INFO [SlabPoolCleaner] 2018-12-27 05:07:01,836 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:01,836 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239025519 (11%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:01,853 Memtable.java:347 - Writing Memtable-customers_metrics@2086879045(189.254MiB serialized bytes, 163048 ops, 11%/0% of on/off-heap limit) INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:02,398 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140634-Data.db (21.863MiB) for commitlog position ReplayPosition(segmentId=1525681566111, position=14599365) INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:02,912 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140635-Data.db (21.556MiB) for commitlog position ReplayPosition(segmentId=1525681566117, position=19478541) INFO [SlabPoolCleaner] 2018-12-27 05:07:03,034 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:03,035 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239127661 (11%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:03,047 Memtable.java:347 - Writing Memtable-customers_metrics@962624348(189.403MiB serialized bytes, 163122 ops, 11%/0% of on/off-heap limit) INFO [SlabPoolCleaner] 2018-12-27 05:07:04,113 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:04,113 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239049872 (11%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:04,123 Memtable.java:347 - Writing Memtable-customers_metrics@219594445(189.231MiB serialized bytes, 163174 ops, 11%/0% of on/off-heap limit) INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:04,210 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140636-Data.db (21.873MiB) for commitlog position ReplayPosition(segmentId=1525681566123, position=24516704) INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:05,168 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140637-Data.db (21.613MiB) for commitlog position ReplayPosition(segmentId=1525681566129, position=29370537) ERROR [SharedPool-Worker-1] 2018-12-27 05:07:10,349 QueryMessage.java:135 - Unexpected error during query com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-2] 2018-12-27 05:07:10,349 QueryMessage.java:135 - Unexpected error during query com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-4] 2018-12-27 05:07:10,349 QueryMessage.java:135 - Unexpected error during query com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-5] 2018-12-27 05:07:10,350 QueryMessage.java:135 - Unexpected error during query com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-1] 2018-12-27 05:07:10,350 ErrorMessage.java:251 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-2] 2018-12-27 05:07:10,350 ErrorMessage.java:251 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-4] 2018-12-27 05:07:10,351 ErrorMessage.java:251 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-5] 2018-12-27 05:07:10,351 ErrorMessage.java:251 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted INFO [SlabPoolCleaner] 2018-12-27 05:07:13,204 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.33/0.00, live: 0.33/0.00, flushing: 0.00/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:13,204 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239164849 (11%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:13,216 Memtable.java:347 - Writing Memtable-customers_metrics@1858853610(189.322MiB serialized bytes, 163202 ops, 11%/0% of on/off-heap limit) INFO [SlabPoolCleaner] 2018-12-27 05:07:13,801 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:13,801 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239050988 (11%) on-heap, 0 (0%) off-heap Best Rabii --000000000000bb8b6d057fbc350c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

We have some Timeout frequently in our node cassa= ndra, can any one help me to analyse what can be the cause.

<= /div>

INFO=C2=A0 [SlabPoolCleaner] 2018-12-27 05:07= :01,836 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace=3D'= db_cass_agregats', ColumnFamily=3D'customers_metrics') to free = up room. Used total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this:= 0.11/0.11
INFO=C2=A0 [SlabPoolCleaner] 2018-12-27 05:07:01,836 C= olumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239025519= (11%) on-heap, 0 (0%) off-heap
INFO=C2=A0 [MemtableFlushWriter:5= 5844] 2018-12-27 05:07:01,853 Memtable.java:347 - Writing Memtable-customer= s_metrics@2086879045(189.254MiB serialized bytes, 163048 ops, 11%/0% of on/= off-heap limit)
INFO=C2=A0 [MemtableFlushWriter:55845] 2018-12-27= 05:07:02,398 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdf= cas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c= 9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140634-Data.db (21.863MiB= ) for commitlog position ReplayPosition(segmentId=3D1525681566111, position= =3D14599365)
INFO=C2=A0 [MemtableFlushWriter:55844] 2018-12-27 05= :07:02,912 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas= /files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc= 7cec/db_cass_agregats-customers_metrics-tmp-ka-140635-Data.db (21.556MiB) f= or commitlog position ReplayPosition(segmentId=3D1525681566117, position=3D= 19478541)
INFO=C2=A0 [SlabPoolCleaner] 2018-12-27 05:07:03,034 Co= lumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace=3D'db_cass_ag= regats', ColumnFamily=3D'customers_metrics') to free up room. U= sed total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11=
INFO=C2=A0 [SlabPoolCleaner] 2018-12-27 05:07:03,035 ColumnFamil= yStore.java:905 - Enqueuing flush of customers_metrics: 239127661 (11%) on-= heap, 0 (0%) off-heap
INFO=C2=A0 [MemtableFlushWriter:55845] 2018= -12-27 05:07:03,047 Memtable.java:347 - Writing Memtable-customers_metrics@= 962624348(189.403MiB serialized bytes, 163122 ops, 11%/0% of on/off-heap li= mit)
INFO=C2=A0 [SlabPoolCleaner] 2018-12-27 05:07:04,113 ColumnF= amilyStore.java:1197 - Flushing largest CFS(Keyspace=3D'db_cass_agregat= s', ColumnFamily=3D'customers_metrics') to free up room. Used t= otal: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11
INFO=C2=A0 [SlabPoolCleaner] 2018-12-27 05:07:04,113 ColumnFamilyStor= e.java:905 - Enqueuing flush of customers_metrics: 239049872 (11%) on-heap,= 0 (0%) off-heap
INFO=C2=A0 [MemtableFlushWriter:55844] 2018-12-2= 7 05:07:04,123 Memtable.java:347 - Writing Memtable-customers_metrics@21959= 4445(189.231MiB serialized bytes, 163174 ops, 11%/0% of on/off-heap limit)<= /div>
INFO=C2=A0 [MemtableFlushWriter:55845] 2018-12-27 05:07:04,210 Me= mtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/d= b_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_= agregats-customers_metrics-tmp-ka-140636-Data.db (21.873MiB) for commitlog = position ReplayPosition(segmentId=3D1525681566123, position=3D24516704)
INFO=C2=A0 [MemtableFlushWriter:55844] 2018-12-27 05:07:05,168 Memta= ble.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/db_c= ass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agr= egats-customers_metrics-tmp-ka-140637-Data.db (21.613MiB) for commitlog pos= ition ReplayPosition(segmentId=3D1525681566129, position=3D29370537)
<= div>ERROR [SharedPool-Worker-1] 2018-12-27 05:07:10,349 QueryMessage.java:1= 35 - Unexpected error during query
com.google.common.util.concurr= ent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cas= sandra.exceptions.ReadTimeoutException: Operation timed out - received only= 2 responses.
at com= .google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-1= 6.0.jar:na]
at com.g= oogle.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na= ]
at com.google.comm= on.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na]
at com.google.common.c= ache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.ja= r:na]
at org.apache.= cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[= apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:= 75) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.authorize= (ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.Cl= ientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.= 13.jar:2.1.13]
at or= g.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[ap= ache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.hasColumnFamilyAc= cess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.cql3.s= tatements.ModificationStatement.checkAccess(ModificationStatement.java:132)= ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.QueryProcessor.processState= ment(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql= 3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13= .jar:2.1.13]
at org.= apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:= 122) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.Message$Dispatcher= .channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13]
<= div> at org.apache.cassandra.tr= ansport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra= -2.1.13.jar:2.1.13]
= at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelIn= boundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
= at io.netty.channel.AbstractCh= annelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:33= 3) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.= access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.j= ar:4.0.23.Final]
at = io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandler= Context.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at java.util.concurrent.Executors$R= unnableAdapter.call(Executors.java:471) [na:1.7.0_75]
at org.apache.cassandra.concurrent.Abstra= ctTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorSe= rvice.java:164) [apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.concurrent.SEPWor= ker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13]
= at java.lang.Thread.run(Thread= .java:745) [na:1.7.0_75]
Caused by: java.lang.RuntimeException: o= rg.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - = received only 2 responses.
= at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache= -cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apa= che-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.AuthenticatedUser.isSuper(Authent= icatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Cassandr= aAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.1= 3.jar:2.1.13]
at org= .apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~= [apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.PermissionsCache$1.load(Permi= ssionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13]
at com.google.common.cache.LocalCac= he$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:= na]
at com.google.co= mmon.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.j= ar:na]
at com.google= .common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[gu= ava-16.0.jar:na]
at = com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guav= a-16.0.jar:na]
... 2= 3 common frames omitted
Caused by: org.apache.cassandra.exception= s.ReadTimeoutException: Operation timed out - received only 2 responses.
at org.apache.cassandr= a.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13= .jar:2.1.13]
at org.= apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java= :146) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.StorageProxy.fetchR= ows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.servi= ce.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.= 13.jar:2.1.13]
at or= g.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apac= he-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements.SelectStatement.execute= (SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.cql3.s= tatements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassan= dra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cas= sandra-2.1.13.jar:2.1.13]
<= /span>... 32 common frames omitted
ERROR [SharedPool-Worker-2] 20= 18-12-27 05:07:10,349 QueryMessage.java:135 - Unexpected error during query=
com.google.common.util.concurrent.UncheckedExecutionException: j= ava.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutExcep= tion: Operation timed out - received only 2 responses.
at com.google.common.cache.LocalCache$Se= gment.get(LocalCache.java:2201) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.get= (LocalCache.java:3934) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(Loc= alCache.java:3938) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache= .get(LocalCache.java:4821) ~[guava-16.0.jar:na]
at org.apache.cassandra.auth.PermissionsCache.g= etPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.1= 3]
at org.apache.cas= sandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar= :2.1.13]
at org.apac= he.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-c= assandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.service.ClientState.ensureHasPermission(Cli= entState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.Client= State.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13]=
at org.apache.cassa= ndra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apac= he-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements.ModificationStatement.c= heckAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.= 1.13]
at org.apache.= cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[a= pache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryPro= cessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.messag= es.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.ja= r:2.1.13]
at org.apa= che.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [= apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.Message$Dispatcher.channe= lRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13]
at io.netty.channel.SimpleChannel= InboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all= -4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannel= Read(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.= 0.23.Final]
at io.ne= tty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandler= Context.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelH= andlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.= 23.Final.jar:4.0.23.Final]
= at java.util.concurrent.Executors$RunnableAdapter.call(Executors.jav= a:471) [na:1.7.0_75]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Fut= ureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra= -2.1.13.jar:2.1.13]
= at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apach= e-cassandra-2.1.13.jar:2.1.13]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.Rea= dTimeoutException: Operation timed out - received only 2 responses.
at org.apache.cassandra.aut= h.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.= auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassand= ra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassa= ndra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAut= horizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Permissions= Cache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.1= 3]
at org.apache.cas= sandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cas= sandra-2.1.13.jar:2.1.13]
<= /span>at com.google.common.cache.LocalCache$LoadingValueReference.loadFutur= e(LocalCache.java:3524) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.load= Sync(LocalCache.java:2317) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.l= ockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache= $Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na]
... 23 common frames omitted
= Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation = timed out - received only 2 responses.
at org.apache.cassandra.service.ReadCallback.get(ReadCal= lback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.AbstractR= eadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.ja= r:2.1.13]
at org.apa= che.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apac= he-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.StorageProxy.readRegular(Storag= eProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.Storage= Proxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandr= a.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apach= e-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(= SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13]
<= span style=3D"white-space:pre-wrap"> at org.apache.cassandra.auth.Au= th.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13]
... 32 common frames omitted=
ERROR [SharedPool-Worker-4] 2018-12-27 05:07:10,349 QueryMessage= .java:135 - Unexpected error during query
com.google.common.util.= concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apa= che.cassandra.exceptions.ReadTimeoutException: Operation timed out - receiv= ed only 2 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[= guava-16.0.jar:na]
a= t com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0= .jar:na]
at com.goog= le.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar= :na]
at com.google.c= ommon.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-= 16.0.jar:na]
at org.= apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java= :72) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.getPermissions(Aut= h.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.au= thorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.ser= vice.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassand= ra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:24= 5) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.hasColumnF= amilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra= .cql3.statements.ModificationStatement.checkAccess(ModificationStatement.ja= va:132) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.QueryProcessor.proce= ssStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13]<= /div>
at org.apache.cassan= dra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra= -2.1.13.jar:2.1.13]
= at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessag= e.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.Message$Dis= patcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13]=
at org.apache.cassa= ndra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-ca= ssandra-2.1.13.jar:2.1.13]
= at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleCh= annelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.Abs= tractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.= java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandle= rContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.2= 3.Final.jar:4.0.23.Final]
<= /span>at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChann= elHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
<= div> at java.util.concurrent.Ex= ecutors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75]
at org.apache.cassandra.concurre= nt.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareE= xecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.concurren= t.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13]
at java.lang.Thread.ru= n(Thread.java:745) [na:1.7.0_75]
Caused by: java.lang.RuntimeExce= ption: org.apache.cassandra.exceptions.ReadTimeoutException: Operation time= d out - received only 2 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) = ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:9= 7) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.AuthenticatedUser.isSuper= (AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.= CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassand= ra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.jav= a:124) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.PermissionsCache$1.lo= ad(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13]
at com.google.common.cache.= LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-1= 6.0.jar:na]
at com.g= oogle.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guav= a-16.0.jar:na]
at co= m.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:22= 80) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195= ) ~[guava-16.0.jar:na]
... 23 common frames omitted
Caused by: org.apache.cassandra.e= xceptions.ReadTimeoutException: Operation timed out - received only 2 respo= nses.
at org.apache.= cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandr= a-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecu= tor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.StoragePr= oxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13]=
at org.apache.cassa= ndra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cass= andra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:12= 47) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements.SelectStateme= nt.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13]<= /div>
at org.apache.cassan= dra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apa= che-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[= apache-cassandra-2.1.13.jar:2.1.13]
... 32 common frames omitted
ERROR [SharedPool-Wo= rker-5] 2018-12-27 05:07:10,350 QueryMessage.java:135 - Unexpected error du= ring query
com.google.common.util.concurrent.UncheckedExecutionEx= ception: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTi= meoutException: Operation timed out - received only 2 responses.
= at com.google.common.cache.Loc= alCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na]
at com.google.common.cache.Local= Cache.get(LocalCache.java:3934) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.get= OrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$LocalLo= adingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na]
at org.apache.cassandra.auth.Permissi= onsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13= .jar:2.1.13]
at org.= apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-= 2.1.13.jar:2.1.13]
a= t org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) = ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.ensureHasPerm= ission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.serv= ice.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.j= ar:2.1.13]
at org.ap= ache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:2= 29) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements.ModificationS= tatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1= .13.jar:2.1.13]
at o= rg.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.jav= a:235) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.QueryProcessor.proces= s(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.transp= ort.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra= -2.1.13.jar:2.1.13]
= at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.j= ava:439) [apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.Message$Dispatc= her.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13]
at io.netty.channel.Sim= pleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) = [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Fi= nal.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChan= nelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
<= div> at io.netty.channel.Abstra= ctChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [nett= y-all-4.0.23.Final.jar:4.0.23.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Exe= cutors.java:471) [na:1.7.0_75]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorS= ervice$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache= -cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:1= 05) [apache-cassandra-2.1.13.jar:2.1.13]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]=
Caused by: java.lang.RuntimeException: org.apache.cassandra.exce= ptions.ReadTimeoutException: Operation timed out - received only 2 response= s.
at org.apache.cas= sandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.= 1.13]
at org.apache.= cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar= :2.1.13]
at org.apac= he.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[ap= ache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.CassandraAuthorizer.authorize(Ca= ssandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.auth.P= ermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13= .jar:2.1.13]
at org.= apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[= apache-cassandra-2.1.13.jar:2.1.13]
at com.google.common.cache.LocalCache$LoadingValueReference= .loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Se= gment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache= $Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na]
at com.google.common.cache.= LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na]
... 23 common frames omitted<= /div>
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: = Operation timed out - received only 2 responses.
at org.apache.cassandra.service.ReadCallback.g= et(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13]
<= span style=3D"white-space:pre-wrap"> at org.apache.cassandra.service= .AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra= -2.1.13.jar:2.1.13]
= at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:13= 97) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.StorageProxy.readRegu= lar(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.servi= ce.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:= 2.1.13]
at org.apach= e.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:27= 1) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements.SelectStatemen= t.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassand= ra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13= ]
... 32 common fram= es omitted
ERROR [SharedPool-Worker-1] 2018-12-27 05:07:10,350 Er= rorMessage.java:251 - Unexpected exception during request
com.goo= gle.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeEx= ception: org.apache.cassandra.exceptions.ReadTimeoutException: Operation ti= med out - received only 2 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCac= he.java:2201) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:39= 34) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) = ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.jav= a:4821) ~[guava-16.0.jar:na]
at org.apache.cassandra.auth.PermissionsCache.getPermissions(Permi= ssionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.ge= tPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.servic= e.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar= :2.1.13]
at org.apac= he.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) = ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.hasAccess(Cli= entState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.Client= State.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13= .jar:2.1.13]
at org.= apache.cassandra.cql3.statements.ModificationStatement.checkAccess(Modifica= tionStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.Quer= yProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1= .13.jar:2.1.13]
at o= rg.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[= apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.messages.QueryMessage.exe= cute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.trans= port.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.= 1.13.jar:2.1.13]
at = org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java= :335) [apache-cassandra-2.1.13.jar:2.1.13]
at io.netty.channel.SimpleChannelInboundHandler.chan= nelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4= .0.23.Final]
at io.n= etty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChanne= lHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.Abstrac= tChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [n= etty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext$8.run= (AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23= .Final]
at java.util= .concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75= ]
at org.apache.cass= andra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(Abstrac= tTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13]=
at org.apache.cassa= ndra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.= jar:2.1.13]
at java.= lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.la= ng.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: = Operation timed out - received only 2 responses.
at org.apache.cassandra.auth.Auth.selectUser(A= uth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.isSuper= user(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Authentic= atedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2= .1.13]
at org.apache= .cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) = ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.PermissionsCache$1.load(Perm= issionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Permi= ssionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar= :2.1.13]
at com.goog= le.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java= :3524) ~[guava-16.0.jar:na]
= at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.j= ava:2317) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(Lo= calCache.java:2280) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(Loca= lCache.java:2195) ~[guava-16.0.jar:na]
... 23 common frames omitted
Caused by: org.ap= ache.cassandra.exceptions.ReadTimeoutException: Operation timed out - recei= ved only 2 responses.
at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~= [apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.AbstractReadExecutor.get(A= bstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.ser= vice.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.= 13.jar:2.1.13]
at or= g.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324)= ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.StorageProxy.read(Storag= eProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements= .SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.1= 3.jar:2.1.13]
at org= .apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.j= ava:224) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.selectUser(Aut= h.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13]
... 32 common frames omitted
ERROR = [SharedPool-Worker-2] 2018-12-27 05:07:10,350 ErrorMessage.java:251 - Unexp= ected exception during request
com.google.common.util.concurrent.= UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassand= ra.exceptions.ReadTimeoutException: Operation timed out - received only 2 r= esponses.
at com.goo= gle.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.= jar:na]
at com.googl= e.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na]
at com.google.common.c= ache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na]
<= div> at com.google.common.cache= .LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na= ]
at org.apache.cass= andra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apac= he-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) = ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.authorize(Cli= entState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.Client= State.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.j= ar:2.1.13]
at org.ap= ache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache= -cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess= (ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.state= ments.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[a= pache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.QueryProcessor.processStatement= (QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13]
<= span style=3D"white-space:pre-wrap"> at org.apache.cassandra.cql3.Qu= eryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar= :2.1.13]
at org.apac= he.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122)= ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.Message$Dispatcher.cha= nnelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.transp= ort.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1= .13.jar:2.1.13]
at i= o.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboun= dHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChanne= lHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [= netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.acce= ss$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4= .0.23.Final]
at io.n= etty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerCont= ext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at java.util.concurrent.Executors$Runna= bleAdapter.call(Executors.java:471) [na:1.7.0_75]
at org.apache.cassandra.concurrent.AbstractTr= acingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorServic= e.java:164) [apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.concurrent.SEPWorker.r= un(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13]
at java.lang.Thread.run(Thread.java= :745) [na:1.7.0_75]
Caused by: java.lang.RuntimeException: org.ap= ache.cassandra.exceptions.ReadTimeoutException: Operation timed out - recei= ved only 2 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cass= andra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-c= assandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.auth.AuthenticatedUser.isSuper(Authenticate= dUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.CassandraAut= horizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.ja= r:2.1.13]
at org.apa= che.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apa= che-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.PermissionsCache$1.load(Permissio= nsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13]
at com.google.common.cache.LocalCache$L= oadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na]<= /div>
at com.google.common= .cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:n= a]
at com.google.com= mon.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-= 16.0.jar:na]
at com.= google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16= .0.jar:na]
... 23 co= mmon frames omitted
Caused by: org.apache.cassandra.exceptions.Re= adTimeoutException: Operation timed out - received only 2 responses.
<= div> at org.apache.cassandra.se= rvice.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar= :2.1.13]
at org.apac= he.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146= ) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.StorageProxy.fetchRows(= StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.S= torageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.j= ar:2.1.13]
at org.ap= ache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-c= assandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.cql3.statements.SelectStatement.execute(Sel= ectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.state= ments.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-= 2.1.13.jar:2.1.13]
a= t org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassand= ra-2.1.13.jar:2.1.13]
... 32 common frames omitted
ERROR [SharedPool-Worker-4] 2018-1= 2-27 05:07:10,351 ErrorMessage.java:251 - Unexpected exception during reque= st
com.google.common.util.concurrent.UncheckedExecutionException:= java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutExc= eption: Operation timed out - received only 2 responses.
at com.google.common.cache.LocalCache$= Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.ge= t(LocalCache.java:3934) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(Lo= calCache.java:3938) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCach= e.get(LocalCache.java:4821) ~[guava-16.0.jar:na]
at org.apache.cassandra.auth.PermissionsCache.= getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.= 13]
at org.apache.ca= ssandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.ja= r:2.1.13]
at org.apa= che.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-= cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.ensureHasPermission(Cl= ientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.Clien= tState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13= ]
at org.apache.cass= andra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apa= che-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements.ModificationStatement.= checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2= .1.13]
at org.apache= .cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[= apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryPr= ocessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.messa= ges.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.j= ar:2.1.13]
at org.ap= ache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) = [apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.Message$Dispatcher.chann= elRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13]
at io.netty.channel.SimpleChanne= lInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-al= l-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChanne= lRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4= .0.23.Final]
at io.n= etty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandle= rContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannel= HandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0= .23.Final.jar:4.0.23.Final]
= at java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja= va:471) [na:1.7.0_75]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Fu= tureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandr= a-2.1.13.jar:2.1.13]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apac= he-cassandra-2.1.13.jar:2.1.13]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.Re= adTimeoutException: Operation timed out - received only 2 responses.
<= div> at org.apache.cassandra.au= th.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra= .auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13]<= /div>
at org.apache.cassan= dra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cass= andra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAu= thorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Permission= sCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.= 13]
at org.apache.ca= ssandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-ca= ssandra-2.1.13.jar:2.1.13]
= at com.google.common.cache.LocalCache$LoadingValueReference.loadFutu= re(LocalCache.java:3524) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loa= dSync(LocalCache.java:2317) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.= lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCach= e$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na]
... 23 common frames omitted
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation= timed out - received only 2 responses.
at org.apache.cassandra.service.ReadCallback.get(ReadCa= llback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.Abstract= ReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.j= ar:2.1.13]
at org.ap= ache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apa= che-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.StorageProxy.readRegular(Stora= geProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.Storag= eProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassand= ra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apac= he-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements.SelectStatement.execute= (SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.auth.A= uth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13]
... 32 common frames omitte= d
ERROR [SharedPool-Worker-5] 2018-12-27 05:07:10,351 ErrorMessag= e.java:251 - Unexpected exception during request
com.google.commo= n.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: = org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -= received only 2 responses.
= at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2= 201) ~[guava-16.0.jar:na]
<= /span>at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[gua= va-16.0.jar:na]
at c= om.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-1= 6.0.jar:na]
at com.g= oogle.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~= [guava-16.0.jar:na]
= at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCac= he.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.getPermissi= ons(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientS= tate.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13]<= /div>
at org.apache.cassan= dra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-= cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.hasAccess(ClientState.= java:245) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.ClientState.has= ColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.= 13]
at org.apache.ca= ssandra.cql3.statements.ModificationStatement.checkAccess(ModificationState= ment.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.QueryProcess= or.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:= 2.1.13]
at org.apach= e.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-c= assandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.transport.messages.QueryMessage.execute(Que= ryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.transport.Mes= sage$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar= :2.1.13]
at org.apac= he.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [a= pache-cassandra-2.1.13.jar:2.1.13]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(= SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Fi= nal]
at io.netty.cha= nnel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandler= Context.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannel= HandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all= -4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext$8.run(Abstrac= tChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]<= /div>
at java.util.concurr= ent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75]
<= div> at org.apache.cassandra.co= ncurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracing= AwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.con= current.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.= 13]
at java.lang.Thr= ead.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.lang.Runti= meException: org.apache.cassandra.exceptions.ReadTimeoutException: Operatio= n timed out - received only 2 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java= :276) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.isSuperuser(Auth.= java:97) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.AuthenticatedUser.i= sSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra= .auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-c= assandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCac= he.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.PermissionsCach= e$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13]
at com.google.common.= cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[g= uava-16.0.jar:na]
at= com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) = ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.j= ava:2280) ~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.jav= a:2195) ~[guava-16.0.jar:na]
... 23 common frames omitted
Caused by: org.apache.cassa= ndra.exceptions.ReadTimeoutException: Operation timed out - received only 2= responses.
at org.a= pache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-ca= ssandra-2.1.13.jar:2.1.13]
= at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractRea= dExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.Stora= geProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1= .13]
at org.apache.c= assandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-= cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.service.StorageProxy.read(StorageProxy.jav= a:1247) ~[apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.cql3.statements.SelectSta= tement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.= 13]
at org.apache.ca= ssandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~= [apache-cassandra-2.1.13.jar:2.1.13]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265= ) ~[apache-cassandra-2.1.13.jar:2.1.13]
... 32 common frames omitted
INFO=C2=A0 [Slab= PoolCleaner] 2018-12-27 05:07:13,204 ColumnFamilyStore.java:1197 - Flushing= largest CFS(Keyspace=3D'db_cass_agregats', ColumnFamily=3D'cus= tomers_metrics') to free up room. Used total: 0.33/0.00, live: 0.33/0.0= 0, flushing: 0.00/0.00, this: 0.11/0.11
INFO=C2=A0 [SlabPoolClean= er] 2018-12-27 05:07:13,204 ColumnFamilyStore.java:905 - Enqueuing flush of= customers_metrics: 239164849 (11%) on-heap, 0 (0%) off-heap
INFO= =C2=A0 [MemtableFlushWriter:55845] 2018-12-27 05:07:13,216 Memtable.java:34= 7 - Writing Memtable-customers_metrics@1858853610(189.322MiB serialized byt= es, 163202 ops, 11%/0% of on/off-heap limit)
INFO=C2=A0 [SlabPool= Cleaner] 2018-12-27 05:07:13,801 ColumnFamilyStore.java:1197 - Flushing lar= gest CFS(Keyspace=3D'db_cass_agregats', ColumnFamily=3D'custome= rs_metrics') to free up room. Used total: 0.45/0.00, live: 0.33/0.00, f= lushing: 0.11/0.00, this: 0.11/0.11
INFO=C2=A0 [SlabPoolCleaner] = 2018-12-27 05:07:13,801 ColumnFamilyStore.java:905 - Enqueuing flush of cus= tomers_metrics: 239050988 (11%) on-heap, 0 (0%) off-heap

Best=C2=A0
Rabii
--000000000000bb8b6d057fbc350c--