drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jinfengni <...@git.apache.org>
Subject [GitHub] drill issue #905: DRILL-1162: Fix OOM for hash join operator when the right ...
Date Sun, 10 Sep 2017 06:03:24 GMT
Github user jinfengni commented on the issue:

    This OOM problem exposes two problems. The first one is in planning time, where we choose
a sub-optimal plan, due to the inaccurate estimation of row count because of missing of appropriate
statistics. The second one is in execution time, where we may need understand whether Drill
uses too much memory and whether spill to disk is an option. I think the two are complementary
to each other; even when we have spill to disk for hash join, if planner choose a sub-optimal
plan, the query still could take long, long time to complete.
    Looks like the PR is addressing the 1st issue. I agree that the root cause is row count
estimation, which is more appropriate to defer to the enhancement of statistics support. For
swapJoin logic, the proposal of getMaxRowCount() seems to be in the line of adjusting row
count estimation. I like better the idea of combining row count + column count, which was
essentially adopted in swapInput() by LoptJoinOptmizeRule.
    For HashTable build side cost, hash table only has to hold the join key. However, since
hash join is a blocking operator, it has hold all the records in the build side, meaning total
memory requirement (for both hash table + non-join key columns) depends on row count and column
count. Therefore, the cost model of hash join should reflect that. Can we use similar idea
in SwapHashJoinVisitor?
    One further improvement would be to modify HashJoinPrel.computeSelfCost(). Today we only
consider join key width, and it makes sense to adjust that logic, by considering the total
column counts in build side. Such logic could be extracted into a common places, then SwapHashJoinVisitor
could call the same shared logic to decide whether it's cost-wise optimal to swap the input
sides. Thoughts?


View raw message