incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Senthil, Athinanthny X. -ND" <Athinanthny.X.Senthil....@disney.com>
Subject socket timeout errors in one DC in a multi DC cluster
Date Mon, 03 Feb 2014 23:37:23 GMT

Experiencing socket timeout errors in one  DC in most of the nodes in multi dc cluster. Here
is error. Client is having intermittent high response time issues in this DC.     DC1 does
not experience any timeout issues, but DC2 does though.  This error started occurring recently
and repeats consecutive days. Any suggestions on cause for it?

During this situation, when we try to run queries on CQL3 on local server itself, sometime
we get rpc_timeout errors. But this is intermittent as well.


ERROR [Thrift:97]   CustomTThreadPoolServer.java (line 219) Error occurred during processing
of message.
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only
2 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
at org.apache.cassandra.service.ClientState.authorize(ClientState.java:308)
at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:178)
at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:171)
at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:155)
at org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:681)
at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:749)
at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:931)
at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3622)
at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3610)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)



Mime
View raw message