impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Behm (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-3528: Transfer scratch tuple memory in Close() of Parquet scanner.
Date Thu, 12 May 2016 01:58:07 GMT
Alex Behm has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/3041

Change subject: IMPALA-3528: Transfer scratch tuple memory in Close() of Parquet scanner.
......................................................................

IMPALA-3528: Transfer scratch tuple memory in Close() of Parquet scanner.

The lifetime of a scanner thread is decoupled from that of row batches that
it produces. That means that all resources associated with row batches
produced by the scanner thread should be transferred to those batches.

The bug was that we were not transferring the ownership of memory from the
scratch batch to the final row batch returned in HdfsParquetScanner::Close().

Triggering an event that would cause the freed memory to be dereferenced is
possible, but very difficult. My understanding is that it is only possible
in exceptional non-deterministic scenarios, e.g., a query is cancelled just
at the right time, or the scanner hits a parse/decoding error.

Testing: I tested this change locally by running the scanner and nested
types test as well as TPCH, nested TPCH, and TPC-DS.

Change-Id: Ic34d32c9a41ea66b2b2d8f5e187cc84d4cb569b2
---
M be/src/exec/hdfs-parquet-scanner.cc
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/41/3041/1
-- 
To view, visit http://gerrit.cloudera.org:8080/3041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic34d32c9a41ea66b2b2d8f5e187cc84d4cb569b2
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Alex Behm <alex.behm@cloudera.com>

Mime
View raw message