quickstep-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hbdeshmukh <...@git.apache.org>
Subject [GitHub] incubator-quickstep pull request #121: Support for performing partitioned ag...
Date Sun, 23 Oct 2016 00:33:20 GMT
GitHub user hbdeshmukh opened a pull request:

    https://github.com/apache/incubator-quickstep/pull/121

    Support for performing partitioned aggregation.

    - Used for creating a pool of hash tables such that each hash table
      belongs to a unique partition.
    - The partitioning is done on the group-by keys.
    - Wrote a utility function to compute composite hash of a group of
      TypedValues.
    - The Tuple class now supports a method to compute the hash of the entire
      tuple (i.e. hash key is the composite key made up of all the
      attributes in the tuple).
    - Added a check for whether the aggregation is partitioned or not.
    - The conditions for whether the aggregation can be partitioned
      are as follows:
      1. The query has a GROUP BY clause.
      2. There are no aggrgeations with a DISTINCT clause.
      3. The estimated number of groups are greater than a pre-defined
      threshold.
      4. The query has at least one aggregation function.
    - Method for partitioned aggregation with GROUP BY
    - StorageBlock now provides a method for performing GROUP BY aggregation
      in a partitioned way.
    - AggregationOperationState calls appropriate method (i.e.
      aggregateGroupBy or aggregateGroupByPartitioned) based on the way in
      which aggregation is being performed.
    
    Performance comparison: TPC-H SF100. 
    
    **Query**|**master**|**partitioned aggregation**|**Speedup over master**
    :-----:|:-----:|:-----:|:-----:
    1|16,046|14,625|1.10
    2|5,625|3,316|1.70
    3|6,861|7,987|0.86
    4|2,662|5,149|0.52
    5|4,364|4,586|0.95
    6|398|397|1.00
    7|23,367|24,600|0.95
    8|3,274|3,701|0.88
    9|10,050|13,029|0.77
    10|15,296|9,502|1.61
    11|2,110|931|2.27
    12|1,805|3,206|0.56
    13|34,220|17,551|1.95
    14|771|815|0.95
    15|4,435|915|4.84
    16|8,661|8,489|1.02
    17|160,707|18,885|8.51
    18|66,309|27,744|2.39
    19|1,475|1,566|0.94
    20|55,381|11,429|4.85
    21|121,310|127,343|0.95
    22|6,792|6,683|1.02
    Total|551,921|312,448|1.77
    
    Some queries e.g. 13, 15, 17, 18, 20 see drastic improvement, primarily because the FinalizeAggregation
operation is multi-threaded, compared to the current single threaded implementation. Overall,
we see a 1.77x improvement on the entire workload. 
    
    For some queries we see significant degradation e.g. 4, 9 and 12. I looked at 9 and 12's
individual runs and it seemed mostly an issue of large variance between the runs. For Q4,
I saw a consistent degradation. So far, I couldn't figure out its reason.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/incubator-quickstep partitioned-aggregate-new

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-quickstep/pull/121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #121
    
----
commit 4045f192e19bdba3fa2faa6f49a47ce2bd7f48a6
Author: Harshad Deshmukh <hbdeshmukh@apache.org>
Date:   2016-09-21T16:43:39Z

    Support for performing partitioned aggregation.
    
    - Used for creating a pool of hash tables such that each hash table
      belongs to a unique partition.
    - The partitioning is done on the group-by keys.
    - Wrote a utility function to compute composite hash of a group of
      TypedValues.
    - Added a check for whether the aggregation is partitioned or not.
    - The conditions for whether the aggregation can be partitioned
      are as follows:
      1. The query has a GROUP BY clause.
      2. There are no aggrgeations with a DISTINCT clause.
      3. The estimated number of groups are greater than a pre-defined
      threshold.
      4. The query has at least one aggregation function.
    - Method for partitioned aggregation with GROUP BY
    - StorageBlock now provides a method for performing GROUP BY aggregation
      in a partitioned way.
    - The Tuple class now supports a method to compute the hash of the entire
      tuple (i.e. hash key is the composite key made up of all the
      attributes in the tuple).
    - AggregationOperationState calls appropriate method (i.e.
      aggregateGroupBy or aggregateGroupByPartitioned) based on the way in
      which aggregation is being performed.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message