Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E1371200B4B for ; Thu, 7 Jul 2016 02:08:26 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id DFC9F160A73; Thu, 7 Jul 2016 00:08:26 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 30CD3160A64 for ; Thu, 7 Jul 2016 02:08:26 +0200 (CEST) Received: (qmail 31214 invoked by uid 500); 7 Jul 2016 00:08:25 -0000 Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@impala.incubator.apache.org Delivered-To: mailing list dev@impala.incubator.apache.org Received: (qmail 31203 invoked by uid 99); 7 Jul 2016 00:08:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jul 2016 00:08:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C0B63CA86A for ; Thu, 7 Jul 2016 00:08:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.362 X-Spam-Level: X-Spam-Status: No, score=0.362 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001] autolearn=disabled Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id egRBB7JYib3T for ; Thu, 7 Jul 2016 00:08:22 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 1D8F860CCF for ; Thu, 7 Jul 2016 00:08:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id u6708KAQ006830; Thu, 7 Jul 2016 00:08:20 GMT Message-Id: <201607070008.u6708KAQ006830@ip-10-146-233-104.ec2.internal> Date: Thu, 7 Jul 2016 00:08:20 +0000 From: "Sailesh Mukil (Code Review)" To: impala-cr@cloudera.com, dev@impala.incubator.apache.org CC: Dan Hecht , Henry Robinson Reply-To: sailesh@cloudera.com X-Gerrit-MessageType: abandon Subject: =?UTF-8?Q?=5BImpala-CR=5D=28cdh5-trunk=29_IMPALA-2988=3A_Refactor_HdfsTableSink=3A=3AClose=28=29_so_that_it_cannot_fail=0A?= X-Gerrit-Change-Id: I87afc6def8dc5bac313928a492e251782500e305 X-Gerrit-ChangeURL: X-Gerrit-Commit: 845f916b9b5414fb294c6116ccad586574b67868 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.2 archived-at: Thu, 07 Jul 2016 00:08:27 -0000 Sailesh Mukil has abandoned this change. Change subject: IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail ...................................................................... Abandoned Looked into this again. The problem with moving ClosePartitionFile() from Close() to FlushFinal() is that if a query is cancelled, FlushFinal() wouldn't get called, and we'd end up with a lot of leaked file handles. So the only work around is to call ClosePartitionFile() in FlushFinal() and then check again in HdfsTableSink->Close() if the files are closed or not, and call ClosePartitionFile() on the remaining open files. This is wasteful and on tables with a lot of partitions (e.g. >1000), we'd end up looping over the partitions twice on the teardown path. Also, if ClosePartitionFile() fails, there's no action we can take other than report that error, which would mean that the file handles would probably still be leaked. So opting to leave ClosePartitionFile() as it is in HdfsTableSink->Close() and leave FlushFinal() blank. -- To view, visit http://gerrit.cloudera.org:8080/2678 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I87afc6def8dc5bac313928a492e251782500e305 Gerrit-PatchSet: 2 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Sailesh Mukil