impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Behm (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-5310: Part 2: Add SAMPLED NDV() function.
Date Tue, 12 Dec 2017 18:07:33 GMT
Hello Bharath Vissapragada, Jim Apple, Dan Hecht, 

I'd like you to reexamine a change. Please visit

to look at the new patch set (#11).

Change subject: IMPALA-5310: Part 2: Add SAMPLED_NDV() function.

IMPALA-5310: Part 2: Add SAMPLED_NDV() function.

Adds a new SAMPLED_NDV() aggregate function that is
intended to be used in COMPUTE STATS TABLESAMPLE.
This patch only adds the function itself. Integration
with COMPUTE STATS will come in a separate patch.

SAMPLED_NDV() estimates the number of distinct values (NDV)
based on a sample of data and the corresponding sampling rate.
The main idea is to collect several x/y data points where x is
the number of rows and y is the corresponding NDV estimate.
These data points are used to fit an objective function to the
data such that the true NDV can be extrapolated.
The aggregate function maintains a fixed number of HyperLogLog
intermediates to compute the x/y points.
Several objective functions are fit and the best-fit one is
used for extrapolation.

Adds the MPFIT C library to perform curve fitting:

The library is a C port from Fortran. Scipy uses the
Fortran version of the library for curve fitting.

- added functional tests
- core/hdfs run passed

Change-Id: Ia51d56ee67ec6073e92f90bebb4005484138b820
M be/src/exprs/
M be/src/exprs/aggregate-functions.h
A be/src/thirdparty/mpfit/DISCLAIMER
A be/src/thirdparty/mpfit/README
A be/src/thirdparty/mpfit/mpfit.c
A be/src/thirdparty/mpfit/mpfit.h
M be/src/util/CMakeLists.txt
A be/src/util/
A be/src/util/mpfit-util.h
M bin/rat_exclude_files.txt
M bin/
M fe/src/main/java/org/apache/impala/analysis/
M fe/src/main/java/org/apache/impala/catalog/
M fe/src/main/java/org/apache/impala/catalog/
M fe/src/test/java/org/apache/impala/analysis/
M fe/src/test/java/org/apache/impala/common/
M tests/query_test/
19 files changed, 3,976 insertions(+), 12 deletions(-)

  git pull ssh:// refs/changes/69/8569/11
To view, visit
To unsubscribe, visit

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia51d56ee67ec6073e92f90bebb4005484138b820
Gerrit-Change-Number: 8569
Gerrit-PatchSet: 11
Gerrit-Owner: Alex Behm <>
Gerrit-Reviewer: Alex Behm <>
Gerrit-Reviewer: Bharath Vissapragada <>
Gerrit-Reviewer: Dan Hecht <>
Gerrit-Reviewer: Jim Apple <>

  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message