impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henry Robinson (Code Review)" <>
Subject [Impala-CR](cdh5-trunk) IMPALA-3141: Send full filters when filter production is disabled
Date Tue, 15 Mar 2016 17:23:40 GMT
Henry Robinson has uploaded a new patch set (#5).

Change subject: IMPALA-3141: Send full filters when filter production is disabled

IMPALA-3141: Send full filters when filter production is disabled

The PHJ may disable runtime filter production for one of several
reasons, including a predicted high false-positive rate. If the filters
are not produced, any scans will wait for their entire timeout before

This patch changes the filter logic to always send a filter, even if one
wasn't actually produced by the PHJ. To preserve correctness, that
filter must contain every element of the set. Such a 'full' filter is
represented by (BloomFilter*)NULL. This allows us to make no changes to
RuntimeFilter::Eval(), which already returns true if the member Bloom
filter is NULL.

In RPCs, a new field is added to TBloomFilter to identify full filters.

The HdfsParquetScanner checks to see if filters are full, and disables
them if so.

There is some miscellaneous cleanup in this patch, particularly the
removal of unused members in BloomFilter.

This patch has been manually tested on queries that would otherwise take
a long time to time-out. A unit test will be added to
runtime_filters_test once the review of that patch has been
completed (and so this patch won't go in until that point).

Change-Id: I04b3e6542651c1e7b77a9bab01d0e3d9506af42f
M be/src/exec/
M be/src/exec/blocking-join-node.h
M be/src/exec/
M be/src/exec/
M be/src/exec/
M be/src/exec/
M be/src/exec/partitioned-hash-join-node.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/runtime-filter.h
M be/src/util/
M be/src/util/
M be/src/util/bloom-filter.h
M be/src/util/
M be/src/util/cpu-info.h
M common/thrift/ImpalaInternalService.thrift
16 files changed, 197 insertions(+), 165 deletions(-)

  git pull ssh:// refs/changes/75/2475/5
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04b3e6542651c1e7b77a9bab01d0e3d9506af42f
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Henry Robinson <>
Gerrit-Reviewer: Marcel Kornacker <>

View raw message