ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IGNITE-7736) SQL COPY: streaming model for network packets instead of request-response model
Date Fri, 16 Feb 2018 10:59:00 GMT
Vladimir Ozerov created IGNITE-7736:
---------------------------------------

             Summary: SQL COPY: streaming model for network packets instead of request-response
model
                 Key: IGNITE-7736
                 URL: https://issues.apache.org/jira/browse/IGNITE-7736
             Project: Ignite
          Issue Type: Task
            Reporter: Vladimir Ozerov
             Fix For: 2.5


*Problem*
Currently data transfer in COPY command is implemented as a series of request-responses. When
request is received, it is parsed synchronously and passed to the streamer, then response
is sent. This is not very efficient approach:
# We hardly could utilize long fat network channels efficiently as we spend a lot of time
waiting for a very small response (ack).
# Parsing takes and adding data to the streamer takes time (especially if we reached streamer
buffer limitations and are blocked waiting for responses from data nodes). During this period
network is not utilized and file data is not transferred further.

*Solution*
Let's fix the problem iteratively as follows:
# Introduce asynchrony - when request is received, send the response immediately before data
processing
# Then consider sending one ack for several requests instead of sending ack for every request
# When multiple simultaneous requests are enabled (previous point), consider moving data processing
to separate stream, so that we can read data from the socket as fast as possible 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message