ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-7526) SQL: Introduce memory region for reducer merge results with disk offload
Date Wed, 13 Jun 2018 10:28:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510929#comment-16510929

Vladimir Ozerov commented on IGNITE-7526:

Looks like we can try to piggyback on H2 external storage as follows:
1) Make our H2 instance persistable and allow to set max memory rows; this way H2 will store
intermediate results on disk
2) Make our merge table use the same infrastructure to get rid of OOME on a client
3) Create a patch for H2 to offload GROUP BY results to disk

> SQL: Introduce memory region for reducer merge results with disk offload
> ------------------------------------------------------------------------
>                 Key: IGNITE-7526
>                 URL: https://issues.apache.org/jira/browse/IGNITE-7526
>             Project: Ignite
>          Issue Type: Task
>          Components: sql
>            Reporter: Vladimir Ozerov
>            Assignee: Taras Ledkov
>            Priority: Major
> Currently all results received from map nodes are stored inside reducer's heap memory.
What is worse, in case of complex queries, such as having sorts or groupings, need to collect
all results from mappers first before final processing could be applied. In case of big results
set (or intermediate results) this could easily lead to OOME on reducer. 
> To mitigate this we should introduce special memory area where intermediate results could
be stored. All final processing should be stored in the same area as well. This area should
be of limited size and should be able to offload results to disk in case of overflow.
> We could start with our B+Tree and free list and store results in some K-V form. 

This message was sent by Atlassian JIRA

View raw message