impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <>
Subject [Impala-CR](cdh5-trunk) IMPALA-3780: avoid many small reads past end of block
Date Thu, 30 Jun 2016 06:37:15 GMT
Hello Dan Hecht,

I'd like you to reexamine a change.  Please visit

to look at the new patch set (#5).

Change subject: IMPALA-3780: avoid many small reads past end of block

IMPALA-3780: avoid many small reads past end of block

The text scanner had some pathological behaviour when reading
significantly past the end of it scan range. E.g. reading a 256mb string
that's split across blocks. ScannerContext defaulted to issuing 1kb
reads, even if the scan node requested significantly more data. E.g. if
the Parquet scanner called ReadBytes(16mb), this was chopped up into
1kb reads, which were reassembled in boundary_buffer_.

Increase the minimum read size in this case to 64kb. Reading that amount
of data should not have any significant overhead even if we only read
a few bytes past the end of the scan range.

ScannerContext implements a saner default algorithm that will work better
if scanners make many small reads: it starts with 64kb reads and doubles
the size of each successive read past the end of the scan range. We
also correct pass the 'read_past_size' into GetNextBuffer(), so that
we always read the right amount of data.

Also save some time by pre-sizing the boundary buffer to the correct
size instead of reallocating it multiple times.

Add test case that exercises the code paths for very large strings.

The queries in the test case are vastly faster than before. E.g. 0.6s
vs ~60s for the count(*) query.

Change-Id: Id90c5dea44f07dba5dd465cf325fbff28be34137
M be/src/exec/
M be/src/exec/hdfs-text-scanner.h
M be/src/exec/
M be/src/exec/scanner-context.h
M be/src/runtime/string-buffer.h
M tests/query_test/
6 files changed, 90 insertions(+), 43 deletions(-)

  git pull ssh:// refs/changes/18/3518/5
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id90c5dea44f07dba5dd465cf325fbff28be34137
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Tim Armstrong <>
Gerrit-Reviewer: Dan Hecht <>
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Tim Armstrong <>

View raw message