impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joe McDonnell (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-5039: Fix variability in parquet dictionary filtering test
Date Fri, 10 Mar 2017 21:55:01 GMT
Hello Alex Behm,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/6301

to look at the new patch set (#4).

Change subject: IMPALA-5039: Fix variability in parquet dictionary filtering test
......................................................................

IMPALA-5039: Fix variability in parquet dictionary filtering test

The tests for dictionary filtering look at how many row groups are
processed and how many are filtered by matching text in the profile.
However, the number of row groups processed and filtered by any
individual fragment depends on how the work is split and how many
impalads are running. This causes variability in the test output.

To fix this, the test needs a way to aggregate the results across
fragments. This fix introduces the following syntax for specifying
these aggregates:
aggregate(function_name, field_name): expected_value
This searches the runtime profile for lines that contain
'field_name: number'. It skips the averaged fragment, as this is
derived from all the other fragments.

Currently, only SUM is implemented, and the expected_value is
required to be an integer. It should be easy to implement other
interesting functions like COUNT and MIN/MAX. It would also be
possible to extend it to floats.

Switching the dictionary filtering tests over to this new syntax
eliminates the variability in the tests.

Change-Id: I6b7b84d973b3ac678a24e82900f2637d569158bb
---
D testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-filtering.test
M testdata/workloads/functional-query/queries/QueryTest/parquet-filtering.test
M tests/common/test_result_verifier.py
M tests/query_test/test_mt_dop.py
M tests/query_test/test_scanners.py
5 files changed, 134 insertions(+), 317 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/6301/4
-- 
To view, visit http://gerrit.cloudera.org:8080/6301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b7b84d973b3ac678a24e82900f2637d569158bb
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Joe McDonnell <joemcdonnell@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonnell@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <marcel@cloudera.com>

Mime
View raw message