ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IGNITE-6019) SQL: client node should not hold the whole data set in-memory when possible
Date Thu, 10 Aug 2017 08:07:00 GMT
Vladimir Ozerov created IGNITE-6019:

             Summary: SQL: client node should not hold the whole data set in-memory when possible
                 Key: IGNITE-6019
                 URL: https://issues.apache.org/jira/browse/IGNITE-6019
             Project: Ignite
          Issue Type: Bug
          Components: sql
    Affects Versions: 2.1
            Reporter: Vladimir Ozerov
            Assignee: Alexander Paschenko
             Fix For: 2.2

Our SQL engine requests request data from server nodes in pieces called "page". This allows
us to control memory consumption on client side. However, currently our client code is designed
in a way that all pages are requested from all servers before a single cursor row is returned
to the user. It defeats the whole idea of "cursor" and "page", and could easily crash client
node with OOME. 

We need to fix that and request further pages in a kind of sliding window, keeping no more
than "N" pages in memory simultaneously. Note that sometimes it is not possible, e.g. in case
of {{DISTINCT}} or non-collocated {{GROUP BY}}. In this case we would have to build the whole
result set first anyway. So let's focus on a scenario when the whole result set is not needed.

As currently everything is requested synchronously page-by-page, in the first version it would
be enough to distribute synchronous page requests between cursor reads, without any prefetch.

Implementation details:
1) Optimization should be applied only to {{skipMergeTbl=true}} cases, when complete result
set of map queries is not needed.
2) Starting point is {{GridReduceQueryExecutor#query}}, see {{skipMergeTbl=true}} branch -
this is where we get all pages eagerly.
3) Get no more than one page from the server at a time. We request the page, iterate over
it, then request another page.

This message was sent by Atlassian JIRA

View raw message