Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Thu, 14 Jan 2016 17:23:39 +0000 (UTC)
From: "Jacques Nadeau (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.12924940.1451504701000.111630.1452792219986@Atlassian.JIRA>
In-Reply-To: <JIRA.12924940.1451504701000@Atlassian.JIRA>
References: <JIRA.12924940.1451504701000@Atlassian.JIRA>
 <JIRA.12924940.1451504701051@arcas>
Subject: [jira] [Commented] (DRILL-4236) ExternalSort should use the new
 allocator functionality to better manage it's memory usage
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/DRILL-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098449#comment-15098449 ] 

Jacques Nadeau commented on DRILL-4236:
---------------------------------------

This is a separate bug we should file. The external sort operation needs to reserve any necessary memory and ensure that it doesn't ever hit an out of memory condition.

[~adeneche] & [~sphillips], it seems like we need to go through with a fine tooth comb and ensure that all possible memory allocations within the External sort are correctly handled. It may make sense to break into two possible situations: easily accommodated and hard to accommodate. For the latter, we probably need to create a separate child allocator with a reservation and usage policy that ensures we don't hit out of memory.

> ExternalSort should use the new allocator functionality to better manage it's memory usage
> ------------------------------------------------------------------------------------------
>
>                 Key: DRILL-4236
>                 URL: https://issues.apache.org/jira/browse/DRILL-4236
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Execution - Relational Operators
>    Affects Versions: 1.5.0
>            Reporter: Deneche A. Hakim
>            Assignee: Deneche A. Hakim
>              Labels: documentation
>             Fix For: 1.5.0
>
>
> Before DRILL-4215, the sort operator wasn't able to correctly compute it's memory usage, and so it jumped through a bunch of hoops to try to figure out when it should spill to disk.
> With the transfer accounting in place, this code can be greatly simplified to just use the current operator memory allocation.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)