impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailesh Mukil (Code Review)" <>
Subject [Impala-CR](cdh5-trunk) IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail
Date Thu, 07 Jul 2016 00:08:20 GMT
Sailesh Mukil has abandoned this change.

Change subject: IMPALA-2988: Refactor HdfsTableSink::Close() so that it cannot fail


Looked into this again. The problem with moving ClosePartitionFile() from Close() to FlushFinal()
is that if a query is cancelled, FlushFinal() wouldn't get called, and we'd end up with a
lot of leaked file handles. So the only work around is to call ClosePartitionFile() in FlushFinal()
and then check again in HdfsTableSink->Close() if the files are closed or not, and call
ClosePartitionFile() on the remaining open files. This is wasteful and on tables with a lot
of partitions (e.g. >1000), we'd end up looping over the partitions twice on the teardown

Also, if ClosePartitionFile() fails, there's no action we can take other than report that
error, which would mean that the file handles would probably still be leaked.

So opting to leave ClosePartitionFile() as it is in HdfsTableSink->Close() and leave FlushFinal()

To view, visit
To unsubscribe, visit

Gerrit-MessageType: abandon
Gerrit-Change-Id: I87afc6def8dc5bac313928a492e251782500e305
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Sailesh Mukil <>
Gerrit-Reviewer: Dan Hecht <>
Gerrit-Reviewer: Henry Robinson <>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Sailesh Mukil <>

View raw message