Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1094F200CAE for ; Wed, 21 Jun 2017 16:58:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0F29F160BF0; Wed, 21 Jun 2017 14:58:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5D29B160BD0 for ; Wed, 21 Jun 2017 16:58:11 +0200 (CEST) Received: (qmail 67744 invoked by uid 500); 21 Jun 2017 14:58:08 -0000 Mailing-List: contact issues-help@carbondata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.apache.org Delivered-To: mailing list issues@carbondata.apache.org Received: (qmail 67735 invoked by uid 99); 21 Jun 2017 14:58:08 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jun 2017 14:58:08 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 4FFDEDFAEE; Wed, 21 Jun 2017 14:58:08 +0000 (UTC) From: manishgupta88 To: issues@carbondata.apache.org Reply-To: issues@carbondata.apache.org Message-ID: Subject: [GitHub] carbondata pull request #1074: [CARBONDATA-1212] Memory leak in case of comp... Content-Type: text/plain Date: Wed, 21 Jun 2017 14:58:08 +0000 (UTC) archived-at: Wed, 21 Jun 2017 14:58:12 -0000 GitHub user manishgupta88 opened a pull request: https://github.com/apache/carbondata/pull/1074 [CARBONDATA-1212] Memory leak in case of compaction when unsafe is true Problem: Memory leak in case of compaction when unsafe is true Analysis: In case of compaction, queryExecutor object is formed for multiple blocks but the objects are not retained and finish method is called only on the last instance created for query executor. Due to this the memory allocated to precious objects is not released which can lead to out of memory issue. Fix: Add all the object to a list and call finish method on all the objects so that memory gets released Proof: Files attached for memory allocation and release count before and after code changes during compaction. [Compaction_result_after_code_change.txt](https://github.com/apache/carbondata/files/1091871/Compaction_result_after_code_change.txt) [Compaction_result_before_code_change.txt](https://github.com/apache/carbondata/files/1091872/Compaction_result_before_code_change.txt) You can merge this pull request into a Git repository by running: $ git pull https://github.com/manishgupta88/incubator-carbondata compaction_memory_leak_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1074.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1074 ---- commit b13cbcf7fb8c4f0f6968b05877133f60dcb8c270 Author: manishgupta88 Date: 2017-06-21T14:39:12Z Problem: Memory leak in case of compaction when unsafe is true Analysis: In case of compaction, queryExecutor object is formed for multiple blocks but the objects are not retained and finish method is called only on the last instance created for query executor. Due to this the memory allocated to precious objects is not released which can lead to out of memory issue. Fix: Add all the object to a list and call finish method on all the objects so that memory gets released ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastructure@apache.org or file a JIRA ticket with INFRA. ---