hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Pullokkaran" <jpullokka...@hortonworks.com>
Subject Re: Review Request 38768: HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats
Date Fri, 25 Sep 2015 20:49:29 GMT


> On Sept. 25, 2015, 6:55 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java, line
316
> > <https://reviews.apache.org/r/38768/diff/1/?file=1084857#file1084857line316>
> >
> >     Also why is this only at the else condition?
> >     Are we assuming that if the requested cols & returned cols size matches
then order is preserved.
> >     
> >     This seems rather shaky assumptions.
> >     Its better to fix stats API to guarantee the order.
> 
> Chaoyu Tang wrote:
>     No, I did not assume that. The reasons I put the logic in this else conditiion are
because:
>     1. only when the requested cols & returned cols size matches, the stats is mapped
to an index and put into the hiveColStatsMap, see following code in updateColStats (line 363
-367)
>     ==
>           if (hiveColStats != null && hiveColStats.size() == nonPartColNamesThatRqrStats.size())
{
>             for (int i = 0; i < hiveColStats.size(); i++) {
>               // the columns in nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats/hiveColStats
>               // are in same order
>               hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), hiveColStats.get(i));
>             }
>           }
>     ==
>     
>     2. For partitioned table, the hiveColStats is already built in the order of nonPartColNamesThatRqrStats.
There is not necessary to reorder it.
>     ==
>               for (String c : nonPartColNamesThatRqrStats) {
>                   ColStatistics cs = stats.getColumnStatisticsFromColName(c);
>                   if (cs != null) {
>                     hiveColStats.add(cs);
>                   } else {
>                     colNamesFailedStats.add(c);
>                   }
>                }
>     ==
>     
>     3. HMS stats API might not be a proper place to enforce the order as I explained
in last comment.

ok


- John


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review100655
-----------------------------------------------------------


On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2015, 6:37 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11964
>     https://issues.apache.org/jira/browse/HIVE-11964
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> The stats in hiveColStats should be ordered in that of the columns in nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats
before being put in the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
>    hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), hiveColStats.get(i));
> }
> 
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata, hiveNonPartitionCols,
nonPartColNamesThatRqrStats); might not return the stats in same order as those in nonPartColNamesThatRqrStats.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 6c0bd25

> 
> Diff: https://reviews.apache.org/r/38768/diff/
> 
> 
> Testing
> -------
> 
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message