cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <SEAN_R_DUR...@homedepot.com>
Subject RE: Experiencing strange disconnect issue
Date Wed, 16 Mar 2016 17:58:58 GMT
Are you using any of the Tuning Policies (https://docs.datastax.com/en/developer/java-driver/2.0/common/drivers/reference/tuningPolicies_c.html)?
It could be that you are hitting some peak load and the driver is not retrying hosts once
they are marked “down.”


Sean Durity – Lead Cassandra Admin
Big DATA Team
For support, create a JIRA<https://portal.homedepot.com/sites/bigdata/Shared%20Documents/Jira%20Hadoop%20Support%20Workflow.pdf>

From: Bo Finnerup Madsen [mailto:bo.gundersen@gmail.com]
Sent: Tuesday, March 15, 2016 5:24 AM
To: user@cassandra.apache.org
Subject: Experiencing strange disconnect issue

Hi,

We are currently trying to convert an existing java web application to use cassandra, and
while most of it works great :) we have a "small" issue.

After some time, we all connectivity seems to be lost and we get the following errors:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query
failed (tried: /10.61.70.107:9042<http://10.61.70.107:9042> (com.datastax.driver.core.exceptions.TransportException:
[/10.61.70.107<http://10.61.70.107>] Connection has been closed), /10.61.70.108:9042<http://10.61.70.108:9042>
(com.datastax.driver.core.exceptions.TransportException: [/10.61.70.108<http://10.61.70.108>]
Connection has been closed))

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query
failed (tried: /10.61.70.107:9042<http://10.61.70.107:9042> (com.datastax.driver.core.exceptions.DriverException:
Timeout while trying to acquire available connection (you may want to increase the driver
number of per-host connections)), /10.61.70.108:9042<http://10.61.70.108:9042> (com.datastax.driver.core.exceptions.TransportException:
[/10.61.70.108<http://10.61.70.108>] Connection has been closed), /10.61.70.110:9042<http://10.61.70.110:9042>
(com.datastax.driver.core.exceptions.TransportException: [/10.61.70.110<http://10.61.70.110>]
Connection has been closed))

The errors persists, and the application needs to be restarted to recover.

At application startup we create a cluster and a session which we reuse through out the application
as pr. the documentation. We don't specify any other options when connecting than the IP's
of the three servers. We are running cassandra 3.0.3 tar ball in EC2 in a cluster of three
machines. The connections are made using v3.0.0 java driver.

I have uploaded the configuration and logs from our cassandra cluster here: https://gist.github.com/anonymous/452e736b401317b5b38d
The issue happend at 00:44:46.

I would greatly appreciate any ideas as to what we are doing wrong to experience this? :)

Thank you in advance!

Yours sincerely,
  Bo Madsen

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is
intended solely for the addressee. Access to this Email by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject to the terms and
conditions expressed in any applicable governing The Home Depot terms of business or client
engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy
and content of this attachment and for any damages or losses arising from any inaccuracies,
errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature,
which may be contained in this attachment and shall not be liable for direct, indirect, consequential
or special damages in connection with this e-mail message or its attachment.
Mime
View raw message