cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sam Tunnicliffe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12103) Cassandra is hang and cqlsh was not able to login with OperationTimeout error
Date Wed, 29 Jun 2016 08:32:17 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354787#comment-15354787
] 

Sam Tunnicliffe commented on CASSANDRA-12103:
---------------------------------------------

bq. then,why it's normal in DC2?we are using 128G memory,20 core CPU in DC1,any advice for
tuning this?

It's hard to say based on the information here (and JIRA is not the right forum for that discussion).
Perhaps the fact that there are 3 times more nodes in DC2 than DC1 is relevant? (Assuming
that your other keyspaces aren't setup with RF=n like {{system_auth}}). As far as tuning goes,
some useful resources are:

https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html
http://blakeeggleston.com/cassandra-tuning-the-jvm-for-read-heavy-workloads.html
http://stackoverflow.com/questions/30207779/optimal-jvm-settings-for-cassandra

As for the OpsCenter agent issue, it's the heap that's exhausted there, so you probably want
to look at tuning that JVM too. Please note though, that the Apache Cassandra JIRA is not
the place to report issues with third party software like OpsCenter. For that, you'll need
to open a support ticket with the vendor.

> Cassandra is hang and cqlsh was not able to login with OperationTimeout error
> -----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12103
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12103
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core, Local Write-Read Paths
>         Environment: centos 6.5 cassandra 2.1.9
>            Reporter: peng xiao
>            Priority: Critical
>         Attachments: system.log.2016-06-28_1257.gz
>
>
> Hi,
> We have two DCs(DC1 and DC2) with DC1 3 nodes and DC2 9 nodes.
> And we experienced a Timeout error today,all applications connected to DC1 were hang
and no response,even cqlsh was not able to log into any node in DC1.
> I restarted the 3 nodes in DC1,the problem was not resolved.
> Then we switched to DC2,then applications back to normal.
> Could you please help to take a look?
> Thanks
> many errors like below:
> ERROR [SharedPool-Worker-43] 2016-06-28 11:58:49,705 Message.java:538 - Unexpected exception
during request; channel = [id: 0x87e315d6, /172.16.10.198:13604 => /172.16.11.13:9042]
> java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
>         at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.auth.Auth.isExistingUser(Auth.java:86) ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.service.ClientState.login(ClientState.java:206) ~[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:82)
~[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
[apache-cassandra-2.1.9.jar:2.1.9]
>         at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0]
>         at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.1.9.jar:2.1.9]
>         at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.9.jar:2.1.9]
>         at java.lang.Thread.run(Thread.java:744) [na:1.8.0]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message