Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
From: Sudheesh Katkam <skatkam@maprtech.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_65E04765-9F26-4C0E-927A-C70F7443616A"
Message-Id: <C142D54B-0E8D-4B02-BFD4-03CBA3A2C0EF@maprtech.com>
Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\))
Subject: Re: epoll disconnections?
Date: Fri, 25 Mar 2016 12:02:12 -0700
References: 
 <CAJrw0ORCWVX2tMevB5fNDVLcGzHcrXtLuRyV4G1BaXVuOPd86g@mail.gmail.com>
 <CANZQJAzhwKFRB1vwL-+ZXpi66z3PM_6ECQdmrBCFTJxeEhZtzA@mail.gmail.com>
To: dev@drill.apache.org
In-Reply-To: 
 <CANZQJAzhwKFRB1vwL-+ZXpi66z3PM_6ECQdmrBCFTJxeEhZtzA@mail.gmail.com>

--Apple-Mail=_65E04765-9F26-4C0E-927A-C70F7443616A
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

@Jacques, let me check the performance test [1] results from the past =
with epoll disabled (or we have to re-run).

@Hanifi, With higher concurrency in performance testing, queries failed =
with this error:

java.io.IOException: syscall:read(...)() failed: Connection reset by =
peer

This is a known issue [2] with the version of Netty that Drill uses. I =
don=E2=80=99t think there is a specific ticket, but a search [3] shows a =
few relevant tickets.

Thank you,
Sudheesh

[1] https://github.com/mapr/drill-perf-test-framework =
<https://github.com/mapr/drill-perf-test-framework>
[2] https://github.com/netty/netty/issues/3539 =
<https://github.com/netty/netty/issues/3539>[3] =
https://issues.apache.org/jira/browse/DRILL-3119?jql=3Dproject%20%3D%20DRI=
LL%20AND%20text%20~%20%22syscall%3Aread%22 =
<https://issues.apache.org/jira/browse/DRILL-3119?jql=3Dproject%20=3D%20DR=
ILL%20AND%20text%20~%20%22syscall:read%22>

> On Mar 25, 2016, at 11:44 AM, Hanifi Gunes <hgunes@maprtech.com> =
wrote:
>=20
> I am wondering what the issue and its manifestation was back then. Do =
we
> have any JIRAs created for this before?
>=20
>=20
> Thanks.
> -Hanifi
>=20
> On Fri, Mar 25, 2016 at 9:57 AM, Jacques Nadeau <jacques@dremio.com> =
wrote:
>=20
>> Hey All,
>>=20
>> If I recall correctly, many months ago Sudheesh discovered that we =
were
>> having instability in RPC connections in some situations due to bugs =
in the
>> epoll implementation that are fixed in a later version of Netty =
(~4.0.31?).
>> At the time, we shelved switching Netty because it also changed the =
memory
>> caching behavior (same thread to all thread) which seemed like a high =
risk
>> change. I thought that as part of this we decided the safest change =
was to
>> disable epoll RPC in our distribution. However, reviewing drill-env, =
it
>> doesn't look like we do this. See here [1].
>>=20
>> Thoughts?
>>=20
>> [1]
>>=20
>> =
https://github.com/apache/drill/blob/master/distribution/src/resources/dri=
ll-env.sh#L19
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>=20


--Apple-Mail=_65E04765-9F26-4C0E-927A-C70F7443616A--