cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bo Finnerup Madsen <bo.gunder...@gmail.com>
Subject Re: Experiencing strange disconnect issue
Date Wed, 16 Mar 2016 20:30:25 GMT
Hi,

I ran another test with the following client setup:
final Cluster cluster = Cluster.builder()
  .addContactPoints(key.getContactPoints())
  .withSocketOptions(new SocketOptions().setKeepAlive(true))
  .withReconnectionPolicy(Policies.defaultReconnectionPolicy())
  .withRetryPolicy(Policies.defaultRetryPolicy())
  .build();

But unfortunately the same issue happened.
I have uploaded the cassandra driver debug log here:
https://gist.github.com/anonymous/db6fa061298018c46954

It all goes south at about line 433 (time 19:46:43) where the client throws
a:
io.netty.handler.codec.DecoderException:
com.datastax.driver.core.exceptions.DriverInternalError: Adjusted frame
length exceeds 268435456: 462591744 - discarded

Any ideas?

ons. 16. mar. 2016 kl. 19.23 skrev Steve Robenalt <srobenalt@highwire.org>:

> Hi Bo,
>
> I would suggest adding:
>
> .withReconnectionPolicy(new ExponentialReconnectionPolicy(1000,30000))
>
> or something similar to your cluster builder.
>
> Steve
>
>
> On Wed, Mar 16, 2016 at 11:18 AM, Bo Finnerup Madsen <
> bo.gundersen@gmail.com> wrote:
>
>> Hi Sean,
>>
>> Thank you for taking the time to answer :)
>> We are using a very vanilla connection, without any sorts of tuning
>> policies. The cluster/session is constructed as follows:
>>                         final Cluster cluster = Cluster.builder()
>>
>> .addContactPoints(key.getContactPoints())
>>                                         .build();
>>                         final Session session =
>> cluster.connect(key.getKeyspace());
>> Perhaps it is too vanilla and we are missing something?
>>
>> Since I posted the question, I have tried downgrading to cassandra
>> v2.1.13 and java driver 2.1.9. But got the same error. So I suspect it is
>> something we are doing wrong.
>>
>>
>>
>> ons. 16. mar. 2016 kl. 18.59 skrev <SEAN_R_DURITY@homedepot.com>:
>>
>>> Are you using any of the Tuning Policies (
>>> https://docs.datastax.com/en/developer/java-driver/2.0/common/drivers/reference/tuningPolicies_c.html)?
>>> It could be that you are hitting some peak load and the driver is not
>>> retrying hosts once they are marked “down.”
>>>
>>>
>>>
>>>
>>>
>>> Sean Durity – Lead Cassandra Admin
>>>
>>> Big DATA Team
>>>
>>> For support, create a JIRA
>>> <https://portal.homedepot.com/sites/bigdata/Shared%20Documents/Jira%20Hadoop%20Support%20Workflow.pdf>
>>>
>>>
>>>
>>> *From:* Bo Finnerup Madsen [mailto:bo.gundersen@gmail.com]
>>> *Sent:* Tuesday, March 15, 2016 5:24 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Experiencing strange disconnect issue
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> We are currently trying to convert an existing java web application to
>>> use cassandra, and while most of it works great :) we have a "small" issue.
>>>
>>>
>>>
>>> After some time, we all connectivity seems to be lost and we get the
>>> following errors:
>>>
>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All
>>> host(s) tried for query failed (tried: /10.61.70.107:9042
>>> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.107]
>>> Connection has been closed), /10.61.70.108:9042
>>> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.108]
>>> Connection has been closed))
>>>
>>>
>>>
>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All
>>> host(s) tried for query failed (tried: /10.61.70.107:9042
>>> (com.datastax.driver.core.exceptions.DriverException: Timeout while trying
>>> to acquire available connection (you may want to increase the driver number
>>> of per-host connections)), /10.61.70.108:9042
>>> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.108]
>>> Connection has been closed), /10.61.70.110:9042
>>> (com.datastax.driver.core.exceptions.TransportException: [/10.61.70.110]
>>> Connection has been closed))
>>>
>>>
>>>
>>> The errors persists, and the application needs to be restarted to
>>> recover.
>>>
>>>
>>>
>>> At application startup we create a cluster and a session which we reuse
>>> through out the application as pr. the documentation. We don't specify any
>>> other options when connecting than the IP's of the three servers. We are
>>> running cassandra 3.0.3 tar ball in EC2 in a cluster of three machines. The
>>> connections are made using v3.0.0 java driver.
>>>
>>>
>>>
>>> I have uploaded the configuration and logs from our cassandra cluster
>>> here: https://gist.github.com/anonymous/452e736b401317b5b38d
>>>
>>> The issue happend at 00:44:46.
>>>
>>>
>>>
>>> I would greatly appreciate any ideas as to what we are doing wrong to
>>> experience this? :)
>>>
>>>
>>>
>>> Thank you in advance!
>>>
>>>
>>>
>>> Yours sincerely,
>>>
>>>   Bo Madsen
>>>
>>> ------------------------------
>>>
>>> The information in this Internet Email is confidential and may be
>>> legally privileged. It is intended solely for the addressee. Access to this
>>> Email by anyone else is unauthorized. If you are not the intended
>>> recipient, any disclosure, copying, distribution or any action taken or
>>> omitted to be taken in reliance on it, is prohibited and may be unlawful.
>>> When addressed to our clients any opinions or advice contained in this
>>> Email are subject to the terms and conditions expressed in any applicable
>>> governing The Home Depot terms of business or client engagement letter. The
>>> Home Depot disclaims all responsibility and liability for the accuracy and
>>> content of this attachment and for any damages or losses arising from any
>>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>>> items of a destructive nature, which may be contained in this attachment
>>> and shall not be liable for direct, indirect, consequential or special
>>> damages in connection with this e-mail message or its attachment.
>>>
>>
>
>
> --
> Steve Robenalt
> Software Architect
> srobenalt@highwire.org <bzavon@highwire.org>
> (office/cell): 916-505-1785
>
> HighWire Press, Inc.
> 425 Broadway St, Redwood City, CA 94063
> www.highwire.org
>
> Technology for Scholarly Communication
>

Mime
View raw message