Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
MIME-Version: 1.0
In-Reply-To: <CA+0=VoWqyGz=PNf=iK3ma_0HKbgen-ncAAnafsFZ6AYB1yJwOQ@mail.gmail.com>
References: <005201d2d31d$6dd98550$498c8ff0$@gridgain.com> <CABuYRcrvM-jpMKaTXQ8Knpwj88bQajnezy4PsC_yH=360m_yTA@mail.gmail.com>
 <010301d2d3a2$7da4a2f0$78ede8d0$@gridgain.com> <CA+eZwrGAL_SFJFp29hDteAbRM32ZOrRpkww2Wui_K95HzXOa9w@mail.gmail.com>
 <CA+0=VoWqyGz=PNf=iK3ma_0HKbgen-ncAAnafsFZ6AYB1yJwOQ@mail.gmail.com>
From: Michael Griggs <michael.griggs@gridgain.com>
Date: Tue, 23 May 2017 22:56:30 +0100
Message-ID: <CAM5Hfv==XokPQmsAY_tCVOvf94kS=_ED19w53qCyAM8QCEdgJw@mail.gmail.com>
Subject: Re: Inefficient approach to executing remote SQL queries
To: dev@ignite.apache.org
Content-Type: multipart/alternative; boundary="001a114d931476e8270550380f95"
archived-at: Tue, 23 May 2017 21:56:39 -0000

--001a114d931476e8270550380f95
Content-Type: text/plain; charset="UTF-8"

The problem here is with the initial opening of connections. With a client
who connects and disconnects quickly, and frequently, a 30-second plus
connection time is not workable.

Mike

On 23 May 2017 6:51 pm, "Dmitriy Setrakyan" <dsetrakyan@apache.org> wrote:

> Why do we turn off the connections, once established? Why not keep them
> open, until an endpoint explicitly closes them?
>
> On Tue, May 23, 2017 at 2:16 AM, Sergi Vladykin <sergi.vladykin@gmail.com>
> wrote:
>
> > Michael,
> >
> > I see your point. I think it must not be too hard to start asynchronously
> > establishing connections to all the needed nodes.
> >
> > I've created respective issue in Jira:
> > https://issues.apache.org/jira/browse/IGNITE-5277
> >
> > Sergi
> >
> > 2017-05-23 11:56 GMT+03:00 Michael Griggs <michael.griggs@gridgain.com>:
> >
> > > Hi Val
> > >
> > > This is precisely my point: it's only a minor optimization until the
> > point
> > > when establishing each connection takes 3-4 seconds, and we establish
> 32
> > of
> > > them in sequence.  At that point it becomes a serious issue: the
> customer
> > > cannot run SQL queries from their development machines without them
> > timing
> > > out once out of every two or three runs.  These kind of problems
> > undermine
> > > confidence in Ignite.
> > >
> > > Mike
> > >
> > >
> > > -----Original Message-----
> > > From: Valentin Kulichenko [mailto:valentin.kulichenko@gmail.com]
> > > Sent: 22 May 2017 19:15
> > > To: dev@ignite.apache.org
> > > Subject: Re: Inefficient approach to executing remote SQL queries
> > >
> > > Hi Mike,
> > >
> > > Generally, establishing connections in parallel could make sense, but
> > note
> > > that in most this would be a minor optimization, because:
> > >
> > >    - Under load connections are established once and then reused. If
> you
> > >    observe disconnections during application lifetime under load, then
> > >    probably this should be addressed first.
> > >    - Actual communication is asynchronous, we use NIO for this. If
> > >    connection already exists, sendGeneric() basically just puts a
> message
> > > into
> > >    a queue.
> > >
> > > -Val
> > >
> > > On Mon, May 22, 2017 at 7:04 PM, Michael Griggs <
> > > michael.griggs@gridgain.com
> > > > wrote:
> > >
> > > > Hi Igniters,
> > > >
> > > >
> > > >
> > > > Whilst diagnosing a problem with a slow query, I became aware of a
> > > > potential issue in the Ignite codebase.  When executing a SQL query
> > > > that is to run remotely, the IgniteH2Indexing#send() method is
> called,
> > > > with a Collection<ClusterNode> as one of its parameters.  This
> > > > collection is iterated sequentially, and ctx.io().sendGeneric() is
> > > > called synchronously for each node.  This is inefficient if
> > > >
> > > >
> > > >
> > > > a)       This is the first execution of a query, and thus TCP
> > connections
> > > > have to be established
> > > >
> > > > b)      The cost of establishing a TCP connection is high
> > > >
> > > >
> > > >
> > > > And optionally
> > > >
> > > >
> > > >
> > > > c)       There are a large number of nodes in the cluster
> > > >
> > > >
> > > >
> > > > In my current situation, developers want to run test queries from
> > > > their code running locally, but connected via VPN to their UAT server
> > > > environment.
> > > > The
> > > > cost of opening a TCP connection is in the multiple seconds, as you
> > > > can see from this Ignite log file snippet:
> > > >
> > > > 2017-05-22 18:29:48,908 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:56924,
> > > > rmtAddr=/10.132.80.3:47100]
> > > >
> > > > 2017-05-22 18:29:52,294 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:56923,
> > > > rmtAddr=/10.132.80.30:47102]
> > > >
> > > > 2017-05-22 18:29:58,659 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:56971,
> > > > rmtAddr=/10.132.80.23:47101]
> > > >
> > > > 2017-05-22 18:30:03,183 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:56972,
> > > > rmtAddr=/10.132.80.21:47100]
> > > >
> > > > 2017-05-22 18:30:06,039 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:56973,
> > > > rmtAddr=/10.132.80.21:47103]
> > > >
> > > > 2017-05-22 18:30:10,828 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:57020,
> > > > rmtAddr=/10.132.80.20:47100]
> > > >
> > > > 2017-05-22 18:30:13,060 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:57021,
> > > > rmtAddr=/10.132.80.29:47103]
> > > >
> > > > 2017-05-22 18:30:22,144 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:57022,
> > > > rmtAddr=/10.132.80.22:47103]
> > > >
> > > > 2017-05-22 18:30:26,513 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:57024,
> > > > rmtAddr=/10.132.80.20:47101]
> > > >
> > > > 2017-05-22 18:30:28,526 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/7.1.14.242:57025,
> > > > rmtAddr=/10.132.80.30:47103]
> > > >
> > > >
> > > >
> > > > Comparing the same code that is executed inside of the UAT
> environment
> > > > (so not using the VPN):
> > > >
> > > > 2017-05-22 18:22:18,102 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/10.175.11.38:53288,
> > > > rmtAddr=/10.175.11.58:47100]
> > > >
> > > > 2017-05-22 18:22:18,105 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/10.175.11.38:45890,
> > > > rmtAddr=/10.175.11.54:47101]
> > > >
> > > > 2017-05-22 18:22:18,108 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/127.0.0.1:47582,
> > > > rmtAddr=/127.0.0.1:47100]
> > > >
> > > > 2017-05-22 18:22:18,111 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/127.0.0.1:45240,
> > > > rmtAddr=/127.0.0.1:47103]
> > > >
> > > > 2017-05-22 18:22:18,114 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/10.175.11.38:46280,
> > > > rmtAddr=/10.175.11.15:47100]
> > > >
> > > > 2017-05-22 18:22:18,118 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/10.132.80.21:51476,
> > > > rmtAddr=/10.132.80.29:47103]
> > > >
> > > > 2017-05-22 18:22:18,120 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/10.132.80.21:56274,
> > > > rmtAddr=pocfd-master1/10.132.80.22:47103]
> > > >
> > > > 2017-05-22 18:22:18,124 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/10.132.80.21:53558,
> > > > rmtAddr=pocfd-ignite1/10.132.80.20:47101]
> > > >
> > > > 2017-05-22 18:22:18,127 INFO [TcpCommunicationSpi] - Established
> > > > outgoing communication connection [locAddr=/10.132.80.21:56216,
> > > > rmtAddr=/10.132.80.30:47103]
> > > >
> > > >
> > > >
> > > > This is a design flaw in the Ignite code, as we are relying on the
> > > > client's network behaving in a particular way (i.e., port opening
> being
> > > very fast).
> > > > We should instead try to mask this potential slowness by establishing
> > > > connections in parallel, and waiting on the results.
> > > >
> > > >
> > > >
> > > > I would like to hear others thoughts and comment before we open a
> JIRA
> > > > to look at this.
> > > >
> > > >
> > > >
> > > > Regards
> > > >
> > > > Mike
> > > >
> > > >
> > >
> > >
> >
>

--001a114d931476e8270550380f95--