quickstep-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jianqiao <...@git.apache.org>
Subject [GitHub] incubator-quickstep pull request #179: QUICKSTEP-70-71 Improve aggregation p...
Date Sun, 05 Feb 2017 05:03:11 GMT
GitHub user jianqiao opened a pull request:

    https://github.com/apache/incubator-quickstep/pull/179

    QUICKSTEP-70-71 Improve aggregation performance

    This PR implements two features that improve aggregation performance:
    1. Adds `CollisionFreeVectorTable` to support specialized high-performance aggregation.
    2. Adds support for aggregation copy elision that we only materialize intermediate results
for non-trivial expressions.
    
    For feature 1, when the group-by attribute is a range-bounded single attribute of `INT`
or `LONG` type. We can use a vector of type `std::vector<std::atomic<StateT>>`
to store the aggregation states, where `StateT` is the aggregation state type (currently restricted
to `LONG` and `DOUBLE`). Then during aggregation, for each tuple, we locate the aggregation
state with the group-by key's value as index to the state vector, and concurrently update
the state with C++'s atomic primitives.
    
    For feature 2, note that the current implementation of aggregation always creates a `ColumnVectorsValueAccessor`
to store the results of ALL the input expressions. However, we can avoid the creation of a
column vector (thus avoiding copying values into the column vector) if the aggregation is
on a simple attribute, e.g. `SUM(x)`. Thus by PR, when performing aggregation we prepare two
input `ValueAccessor`s: one BASE accessor that is created directly from the input relation's
storage block, and one DERIVED accessor that is the temporary result `ColumnVectorsValueAccessor`.
Each aggregation argument may be from the base accessor (meaning that it is a simple attribute)
or from the derived accessor (meaning that it is a non-trivial expression that gets evaluated).
The two accessors are then properly handled in aggregation handles and aggregation hash tables.
    
    **Main changes:**
    `expressions/aggregation`: Updated the aggregation handles to support copy elision. Also
did some cleanups.
    `relational_operators`: Added `InitializeAggregationOperator` to support parallel initialization
of the aggregation state (just `memset` the memory to 0) -- because it takes a relatively
long time to do the initialization with single thread if the aggregation hash table is large.
    `storage`: Added `CollisionFreeVectorTable`. Renamed `FastHashTable` to `PackedPayloadHashTable`,
made it support copy elision, and cleaned up the class to remove unused methods. Refactored
`AggregationOperationState` to support copy elision and support the new aggregation. Moved
aggregation code out of `StorageBlock`.
    
    This PR significantly improves some TPC-H queries' performance. For example, it improves
TPC-H Q18 from ~27.5s to ~3.5s, with scale factor 100 on a cloudlab machine.
    
    Below shows the TPC-H performance (scale factor 100 on a cloudlab machine) with recently
committed optimizations up to this point:
    
    |  **TPCH SF100** | **master (ms)** | **w/ optimizations (ms)** |
    |  ------ | ------ | ------ |
    |  Q01 | 13629 | 11221 |
    |  Q02 | 537 | 460 |
    |  Q03 | 4824 | 4124 |
    |  Q04 | 2185 | 2203 |
    |  Q05 | 5517 | 5282 |
    |  Q06 | 399 | 401 |
    |  Q07 | 18563 | 3456 |
    |  Q08 | 1746 | 899 |
    |  Q09 | 7247 | 5586 |
    |  Q10 | 6745 | 5665 |
    |  Q11 | 1053 | 247 |
    |  Q12 | 1713 | 1698 |
    |  Q13 | 22896 | 15582 |
    |  Q14 | 805 | 745 |
    |  Q15 | 897 | 431 |
    |  Q16 | 9942 | 9158 |
    |  Q17 | 1588 | 1117 |
    |  Q18 | 27459 | 3507 |
    |  Q19 | 1711 | 1609 |
    |  Q20 | 1204 | 1014 |
    |  Q21 | 8671 | 7886 |
    |  Q22 | 6178 | 724 |
    |  **Total** | **145509** | **83016** |

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/incubator-quickstep collision-free-agg

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-quickstep/pull/179.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #179
    
----
commit 68be4a614f55412632f13051295327fecba1fada
Author: Jianqiao Zhu <jianqiao@cs.wisc.edu>
Date:   2017-01-30T20:46:39Z

    - Adds CollisionFreeVectorTable to support specialized fast path aggregation for range-bounded
single integer group-by key.
    - Supports copy elision for aggregation.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message