Return-Path: X-Original-To: apmail-drill-dev-archive@www.apache.org Delivered-To: apmail-drill-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50B4A186D5 for ; Thu, 27 Aug 2015 17:19:15 +0000 (UTC) Received: (qmail 66765 invoked by uid 500); 27 Aug 2015 17:19:14 -0000 Delivered-To: apmail-drill-dev-archive@drill.apache.org Received: (qmail 66708 invoked by uid 500); 27 Aug 2015 17:19:14 -0000 Mailing-List: contact dev-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list dev@drill.apache.org Received: (qmail 66697 invoked by uid 99); 27 Aug 2015 17:19:14 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2015 17:19:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 374A3C0861 for ; Thu, 27 Aug 2015 17:19:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.001 X-Spam-Level: **** X-Spam-Status: No, score=4.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id JtGSoAOto6Rk for ; Thu, 27 Aug 2015 17:19:05 +0000 (UTC) Received: from mail-io0-f173.google.com (mail-io0-f173.google.com [209.85.223.173]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 01F8020752 for ; Thu, 27 Aug 2015 17:19:04 +0000 (UTC) Received: by iodt126 with SMTP id t126so65280269iod.2 for ; Thu, 27 Aug 2015 10:19:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=R1q+2pmbt5+xu20xCsnbgCm3VA3LECeG+oBUOIhbwic=; b=CHBlXPTJkyK2vsgtrszZozxreh+z/9jUMmKagxcql0lOJmqE0zB/pMTvjYFC2jPtWz KORu621yiRf3S45113eQG/CbeSsRQLidp4+HkIcaddwjPTGINGcQI/5fM7vgDPxtZGwU TK1e6LNwzRqAJC+17URDPec7w+KnKJo4txi+3D4EOm2JmOre75FnTwOtILF9hL4/Gjlc HeJKoD+ZJj5iAOeUodcn1aRSTO1tDt1l3YPXuGq+MgPpAgwikrGGDQLyCl2KiRzu1a5n b2A4QTbVLp2i7Y7H48TJX38exiXaRhA/+4nLbk3O1jgX2xuU8OuJXHSgM4CgIOBjmLzk K1DA== X-Gm-Message-State: ALoCoQltba6GIxSqCXcuJafFaP1LgwbjxEydx41NOStIsaJsS9Yuoa/TYn2+wvcoscOZlBvzpe6D MIME-Version: 1.0 X-Received: by 10.107.167.10 with SMTP id q10mr5711972ioe.66.1440695943985; Thu, 27 Aug 2015 10:19:03 -0700 (PDT) Received: by 10.107.158.10 with HTTP; Thu, 27 Aug 2015 10:19:03 -0700 (PDT) X-Originating-IP: [107.133.150.101] In-Reply-To: References: Date: Thu, 27 Aug 2015 10:19:03 -0700 Message-ID: Subject: Re: [What is the purpose of ExternalSort's MAX_SORT_BYTES From: Steven Phillips To: dev@drill.apache.org Content-Type: multipart/alternative; boundary=001a1142a48af89a2b051e4e29fb --001a1142a48af89a2b051e4e29fb Content-Type: text/plain; charset=UTF-8 I think it probably isn't needed anymore. O believe it is a holdover from before spilling was implemented. It doesn't seem to serve any purpose now. On Thu, Aug 27, 2015 at 9:17 AM, Abdel Hakim Deneche wrote: > anyone ? > > On Tue, Aug 25, 2015 at 2:56 PM, Abdel Hakim Deneche < > adeneche@maprtech.com> > wrote: > > > When running a window function query on large datasets, > > increasing planner.memory.max_query_memory_per_node can actually help the > > query not run out of memory. But in some cases this can cause some issues > > (see DRILL-3555 ) > > > > This seems to be caused by a hardcoded limit in ExternalSort called > > MAX_SORT_BYTES. What is the purpose of this limit ? > > > > -- > > > > Abdelhakim Deneche > > > > Software Engineer > > > > > > > > > > Now Available - Free Hadoop On-Demand Training > > < > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > > > -- > > Abdelhakim Deneche > > Software Engineer > > > > > Now Available - Free Hadoop On-Demand Training > < > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > --001a1142a48af89a2b051e4e29fb--