impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailesh Mukil (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail
Date Thu, 07 Jul 2016 00:08:20 GMT
Sailesh Mukil has abandoned this change.

Change subject: IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail
......................................................................


Abandoned

Looked into this again. The problem with moving ClosePartitionFile() from Close() to FlushFinal()
is that if a query is cancelled, FlushFinal() wouldn't get called, and we'd end up with a
lot of leaked file handles. So the only work around is to call ClosePartitionFile() in FlushFinal()
and then check again in HdfsTableSink->Close() if the files are closed or not, and call
ClosePartitionFile() on the remaining open files. This is wasteful and on tables with a lot
of partitions (e.g. >1000), we'd end up looping over the partitions twice on the teardown
path.

Also, if ClosePartitionFile() fails, there's no action we can take other than report that
error, which would mean that the file handles would probably still be leaked.

So opting to leave ClosePartitionFile() as it is in HdfsTableSink->Close() and leave FlushFinal()
blank.

-- 
To view, visit http://gerrit.cloudera.org:8080/2678
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: abandon
Gerrit-Change-Id: I87afc6def8dc5bac313928a492e251782500e305
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Sailesh Mukil <sailesh@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Sailesh Mukil <sailesh@cloudera.com>

Mime
View raw message