Return-Path: X-Original-To: apmail-drill-issues-archive@minotaur.apache.org Delivered-To: apmail-drill-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B2AF917A42 for ; Thu, 9 Apr 2015 01:11:12 +0000 (UTC) Received: (qmail 51216 invoked by uid 500); 9 Apr 2015 01:11:12 -0000 Delivered-To: apmail-drill-issues-archive@drill.apache.org Received: (qmail 51180 invoked by uid 500); 9 Apr 2015 01:11:12 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 51170 invoked by uid 99); 9 Apr 2015 01:11:12 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Apr 2015 01:11:12 +0000 Date: Thu, 9 Apr 2015 01:11:12 +0000 (UTC) From: "Steven Phillips (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (DRILL-2728) Merge spill files in ExternalSort MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Steven Phillips created DRILL-2728: -------------------------------------- Summary: Merge spill files in ExternalSort Key: DRILL-2728 URL: https://issues.apache.org/jira/browse/DRILL-2728 Project: Apache Drill Issue Type: Improvement Reporter: Steven Phillips Assignee: Steven Phillips When external sort spills, it first merges all of the in-memory batches, and then keeps one batch in memory, while spilling the rest to disk. It's possible that we will still run out of memory, due to the single batch held in memory for each spill. When the memory due to these batches becomes too large, we should combine the spill files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)