From dev-return-34811-archive-asf-public=cust-asf.ponee.io@drill.apache.org Wed Jan 10 00:54:13 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 07397180718 for ; Wed, 10 Jan 2018 00:54:13 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id EB937160C3F; Tue, 9 Jan 2018 23:54:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3D318160C17 for ; Wed, 10 Jan 2018 00:54:12 +0100 (CET) Received: (qmail 64912 invoked by uid 500); 9 Jan 2018 23:54:11 -0000 Mailing-List: contact dev-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list dev@drill.apache.org Received: (qmail 64901 invoked by uid 99); 9 Jan 2018 23:54:10 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jan 2018 23:54:10 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id D62D2DFC32; Tue, 9 Jan 2018 23:54:10 +0000 (UTC) From: sachouche To: dev@drill.apache.org Reply-To: dev@drill.apache.org Message-ID: Subject: [GitHub] drill pull request #1087: Attempt to fix memory leak in Parquet Content-Type: text/plain Date: Tue, 9 Jan 2018 23:54:10 +0000 (UTC) GitHub user sachouche opened a pull request: https://github.com/apache/drill/pull/1087 Attempt to fix memory leak in Parquet ** Problem Description ** This is an extremely rare leak which I was able to emulate by putting a sleep in the AsyncPageReader right after reading the page and before enqueue in the result queue. This is how this issue could manifest itself in real life scenario: - AsyncPageReader reads a page into a buffer but didn't enqueue yet the result (thread got preempted) - Parquet Scan thread blocked waiting on the task (Future object dequeued) - Cancel received and Scan thread interrupted - Future.get() returns (Future object is lost) - Scan thread executes release logic - Scan thread is not able to interrupt the AsyncPageReader thread since the future object is lost - AsyncPageReader thread resumes and enqueues the DrillBuf in the result queue - This results in a leak since this buffer is not properly released ** Fix Description ** - The fix is straightforward as we peek the Future object during the blocking get() method - This way, an exception (such as an interrupt) will leave the Future object in the task queue - The cleanup logic will be able to guarantee the DrillBuf object is either GCed by the AsyncPageReader or ParquetScan thread You can merge this pull request into a Git repository by running: $ git pull https://github.com/sachouche/drill DRILL-6079 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/1087.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1087 ---- commit 52030d1d9cc3b8992a10ade8c7126d66e785043a Author: Salim Achouche Date: 2017-12-22T19:50:56Z Attempt to fix memory leak in Parquet ---- ---