hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jesus Camacho Rodriguez (JIRA)" <>
Subject [jira] [Created] (HIVE-11954) Extend logic to choose side table in MapJoin Conversion algorithm
Date Thu, 24 Sep 2015 22:30:05 GMT
Jesus Camacho Rodriguez created HIVE-11954:

             Summary: Extend logic to choose side table in MapJoin Conversion algorithm
                 Key: HIVE-11954
             Project: Hive
          Issue Type: Bug
          Components: Physical Optimizer
    Affects Versions: 2.0.0
            Reporter: Jesus Camacho Rodriguez
            Assignee: Jesus Camacho Rodriguez

Selection of side table (in memory/hash table) in MapJoin Conversion algorithm needs to be
more sophisticated.

In an N way Map Join, Hive should pick an input stream as side table (in memory table) that
has least cost in producing relation (like TS(FIL|Proj)*).

Cost based choice needs extended cost model; without return path its going to be hard to do

For the time being we could employ a modified cost based algorithm for side table selection.

New algorithm is described below:
1. Identify the candidate set of inputs for side table (in memory/hash table) from the inputs
(based on conditional task size)
2. For each of the input identify its cost, memory requirement. Cost is 1 for each heavy weight
relation op (Join, GB, PTF/Windowing, TF, etc.). Cost for an input is the total no of heavy
weight ops in its branch.
3. Order set from #1 on cost & memory req (ascending order)
4. Pick the first element from #3 as the side table.

This message was sent by Atlassian JIRA

View raw message