hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chaoyu Tang" <ctang...@gmail.com>
Subject Re: Review Request 38768: HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats
Date Fri, 25 Sep 2015 20:13:41 GMT


> On Sept. 25, 2015, 6:55 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java, line
316
> > <https://reviews.apache.org/r/38768/diff/1/?file=1084857#file1084857line316>
> >
> >     Also why is this only at the else condition?
> >     Are we assuming that if the requested cols & returned cols size matches
then order is preserved.
> >     
> >     This seems rather shaky assumptions.
> >     Its better to fix stats API to guarantee the order.

No, I did not assume that. The reasons I put the logic in this else conditiion are because:
1. only when the requested cols & returned cols size matches, the stats is mapped to an
index and put into the hiveColStatsMap, see following code in updateColStats (line 363 -367)
==
      if (hiveColStats != null && hiveColStats.size() == nonPartColNamesThatRqrStats.size())
{
        for (int i = 0; i < hiveColStats.size(); i++) {
          // the columns in nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats/hiveColStats
          // are in same order
          hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), hiveColStats.get(i));
        }
      }
==

2. For partitioned table, the hiveColStats is already built in the order of nonPartColNamesThatRqrStats.
There is not necessary to reorder it.
==
          for (String c : nonPartColNamesThatRqrStats) {
              ColStatistics cs = stats.getColumnStatisticsFromColName(c);
              if (cs != null) {
                hiveColStats.add(cs);
              } else {
                colNamesFailedStats.add(c);
              }
           }
==

3. HMS stats API might not be a proper place to enforce the order as I explained in last comment.


- Chaoyu


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review100655
-----------------------------------------------------------


On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2015, 6:37 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11964
>     https://issues.apache.org/jira/browse/HIVE-11964
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> The stats in hiveColStats should be ordered in that of the columns in nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats
before being put in the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
>    hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), hiveColStats.get(i));
> }
> 
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata, hiveNonPartitionCols,
nonPartColNamesThatRqrStats); might not return the stats in same order as those in nonPartColNamesThatRqrStats.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 6c0bd25

> 
> Diff: https://reviews.apache.org/r/38768/diff/
> 
> 
> Testing
> -------
> 
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message