Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <CAO_41X7MWV4v6T1FqDwhwhFLr7Wbov-Judj+TUxqvScpLYWG9g@mail.gmail.com>
References: <CAO_41X7MWV4v6T1FqDwhwhFLr7Wbov-Judj+TUxqvScpLYWG9g@mail.gmail.com>
From: Nate McCall <nate@thelastpickle.com>
Date: Thu, 31 Aug 2017 15:31:07 +1200
Message-ID: <CAKmMYa8C91-PHmeyManBGSVT1VDjQ3Gb3J+f4q_tCwpa1MOVrA@mail.gmail.com>
Subject: Re: Cassandra All host(s) tried for query failed (no host was tried)
To: Cassandra Users <user@cassandra.apache.org>
Content-Type: multipart/alternative; boundary="f403045d9bf678ba2c05580446a8"
archived-at: Thu, 31 Aug 2017 03:31:22 -0000

--f403045d9bf678ba2c05580446a8
Content-Type: text/plain; charset="UTF-8"

If these app instances sit idle for a while, they might just be timing out
their sockets. You can tweak socket settings on the driver as described
here:
https://github.com/datastax/java-driver/tree/3.x/manual/socket_options

Perhaps start with explicitly setting keepAlive to true as that may or may
not be set depending on whether it's using the native epoll extension or
NIO directly (more details about such on the page above).

On Thu, Aug 31, 2017 at 3:10 AM, Ivan Iliev <ivan.iliev.iliev@gmail.com>
wrote:

> Hello everyone,
>
> We are using Cassandra 3.9 for storing quite a lot of data produced from
> our tester machines.
>
> Occasionally, we are seeing issues with apps not being able to communicate
> with Cassandra nodes, returning the following errors (captured in
> servicemix logs):
>
>>  by: com.datastax.driver.core.exceptions.NoHostAvailableException: All
>> host(s) tried for query failed (no host was tried)
>> at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(
>> RequestHandler.java:218)
>> at com.datastax.driver.core.RequestHandler.access$1000(
>> RequestHandler.java:43)
>> at com.datastax.driver.core.RequestHandler$SpeculativeExecution.
>> sendRequest(RequestHandler.java:284)
>> at com.datastax.driver.core.RequestHandler.startNewExecution(
>> RequestHandler.java:115)
>> at com.datastax.driver.core.RequestHandler.sendRequest(
>> RequestHandler.java:91)
>> at com.datastax.driver.core.SessionManager.executeAsync(
>> SessionManager.java:132)
>> ... 107 more
>
>
> As a result, apps that try to send data to cassandra get crashed due to
> running out of memory and we have to restart the containers in which they
> run.
>
> So far I have not been able to identify what might be the cause for this
> as nothing (at least I could not find anything relevant on the timestamps)
> in the cassandra debug and system logs.
>
> Could you share some insight on this ? What to check and where to start
> from , in order to troubleshoot this.
>
> Thanks !
> Ivan
>


-- 
-----------------
Nate McCall
Wellington, NZ
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com

--f403045d9bf678ba2c05580446a8
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">If these app instances sit idle for a while, they might ju=
st be timing out their sockets. You can tweak socket settings on the driver=
 as described here:<div><a href=3D"https://github.com/datastax/java-driver/=
tree/3.x/manual/socket_options">https://github.com/datastax/java-driver/tre=
e/3.x/manual/socket_options</a><br></div><div><br></div><div>Perhaps start =
with explicitly setting keepAlive to true as that may or may not be set dep=
ending on whether it&#39;s using the native epoll extension or NIO directly=
 (more details about such on the page above).=C2=A0</div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, Aug 31, 2017 at 3:1=
0 AM, Ivan Iliev <span dir=3D"ltr">&lt;<a href=3D"mailto:ivan.iliev.iliev@g=
mail.com" target=3D"_blank">ivan.iliev.iliev@gmail.com</a>&gt;</span> wrote=
:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-le=
ft:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hello everyone,<div><b=
r></div><div>We are using Cassandra 3.9 for storing quite a lot of data pro=
duced from our tester machines.</div><div><br></div><div>Occasionally, we a=
re seeing issues with apps not being able to communicate with Cassandra nod=
es, returning the following errors (captured in servicemix logs):</div><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left=
:1px solid rgb(204,204,204);padding-left:1ex"><span style=3D"color:rgb(51,5=
1,51);font-family:Lato,calibri;font-size:14px;background-color:rgb(240,240,=
240)">=C2=A0by: com.datastax.driver.core.<wbr>exceptions.<wbr>NoHostAvailab=
leException: All host(s) tried for query failed (no host was tried)<br></sp=
an><span style=3D"color:rgb(51,51,51);font-family:Lato,calibri;font-size:14=
px;background-color:rgb(240,240,240)">at com.datastax.driver.core.<wbr>Requ=
estHandler.<wbr>reportNoMoreHosts(<wbr>RequestHandler.java:218)<br></span><=
span style=3D"color:rgb(51,51,51);font-family:Lato,calibri;font-size:14px;b=
ackground-color:rgb(240,240,240)">at com.datastax.driver.core.<wbr>RequestH=
andler.access$1000(<wbr>RequestHandler.java:43)<br></span><span style=3D"co=
lor:rgb(51,51,51);font-family:Lato,calibri;font-size:14px;background-color:=
rgb(240,240,240)">at com.datastax.driver.core.<wbr>RequestHandler$<wbr>Spec=
ulativeExecution.<wbr>sendRequest(RequestHandler.<wbr>java:284)<br></span><=
span style=3D"color:rgb(51,51,51);font-family:Lato,calibri;font-size:14px;b=
ackground-color:rgb(240,240,240)">at com.datastax.driver.core.<wbr>RequestH=
andler.<wbr>startNewExecution(<wbr>RequestHandler.java:115)<br></span><span=
 style=3D"color:rgb(51,51,51);font-family:Lato,calibri;font-size:14px;backg=
round-color:rgb(240,240,240)">at com.datastax.driver.core.<wbr>RequestHandl=
er.sendRequest(<wbr>RequestHandler.java:91)<br></span><span style=3D"color:=
rgb(51,51,51);font-family:Lato,calibri;font-size:14px;background-color:rgb(=
240,240,240)">at com.datastax.driver.core.<wbr>SessionManager.executeAsync(=
<wbr>SessionManager.java:132)<br></span><span style=3D"color:rgb(51,51,51);=
font-family:Lato,calibri;font-size:14px;background-color:rgb(240,240,240)">=
... 107 more</span></blockquote><div><span style=3D"color:rgb(51,51,51);fon=
t-family:Lato,calibri;font-size:14px;background-color:rgb(240,240,240)"><br=
></span></div><div>As a result, apps that try to send data to cassandra get=
 crashed due to running out of memory and we have to restart the containers=
 in which they run.<br></div><div><br></div><div>So far I have not been abl=
e to identify what might be the cause for this as nothing (at least I could=
 not find anything relevant on the timestamps) in the cassandra debug and s=
ystem logs.</div><div><br></div><div>Could you share some insight on this ?=
 What to check and where to start from , in order to troubleshoot this.</di=
v><div><br></div><div>Thanks !</div><span class=3D"HOEnZb"><font color=3D"#=
888888"><div>Ivan</div></font></span></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><d=
iv><div dir=3D"ltr"><div><div dir=3D"ltr">-----------------<br>Nate McCall<=
br>Wellington, NZ<br>@zznate<br><br>CTO<br>Apache Cassandra Consulting<br><=
a href=3D"http://www.thelastpickle.com" target=3D"_blank">http://www.thelas=
tpickle.com</a></div></div></div></div></div></div>
</div>

--f403045d9bf678ba2c05580446a8--