Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
 boundary="===============4617478550770594663=="
MIME-Version: 1.0
Subject: Re: Review Request 38768: HIVE-11964: RelOptHiveTable.hiveColStatsMap
 might contain mismatched column stats
From: "Chaoyu Tang" <ctang.ma@gmail.com>
To: "John Pullokkaran" <jpullokkaran@hortonworks.com>,
 "hive" <dev@hive.apache.org>, "Chaoyu Tang" <ctang.ma@gmail.com>
Date: Fri, 25 Sep 2015 20:13:41 -0000
Message-ID: <20150925201341.24659.45514@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Chaoyu Tang" <noreply@reviews.apache.org>
References: <20150925185514.24658.87392@reviews.apache.org>
In-Reply-To: <20150925185514.24658.87392@reviews.apache.org>
Reply-To: "Chaoyu Tang" <ctang.ma@gmail.com>

--===============4617478550770594663==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


> On Sept. 25, 2015, 6:55 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java, line 316
> > <https://reviews.apache.org/r/38768/diff/1/?file=1084857#file1084857line316>
> >
> >     Also why is this only at the else condition?
> >     Are we assuming that if the requested cols & returned cols size matches then order is preserved.
> >     
> >     This seems rather shaky assumptions.
> >     Its better to fix stats API to guarantee the order.

No, I did not assume that. The reasons I put the logic in this else conditiion are because:
1. only when the requested cols & returned cols size matches, the stats is mapped to an index and put into the hiveColStatsMap, see following code in updateColStats (line 363 -367)
==
      if (hiveColStats != null && hiveColStats.size() == nonPartColNamesThatRqrStats.size()) {
        for (int i = 0; i < hiveColStats.size(); i++) {
          // the columns in nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats/hiveColStats
          // are in same order
          hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), hiveColStats.get(i));
        }
      }
==

2. For partitioned table, the hiveColStats is already built in the order of nonPartColNamesThatRqrStats. There is not necessary to reorder it.
==
          for (String c : nonPartColNamesThatRqrStats) {
              ColStatistics cs = stats.getColumnStatisticsFromColName(c);
              if (cs != null) {
                hiveColStats.add(cs);
              } else {
                colNamesFailedStats.add(c);
              }
           }
==

3. HMS stats API might not be a proper place to enforce the order as I explained in last comment.


- Chaoyu


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review100655
-----------------------------------------------------------


On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2015, 6:37 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11964
>     https://issues.apache.org/jira/browse/HIVE-11964
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> The stats in hiveColStats should be ordered in that of the columns in nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats before being put in the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
>    hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), hiveColStats.get(i));
> }
> 
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata, hiveNonPartitionCols, nonPartColNamesThatRqrStats); might not return the stats in same order as those in nonPartColNamesThatRqrStats.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 6c0bd25 
> 
> Diff: https://reviews.apache.org/r/38768/diff/
> 
> 
> Testing
> -------
> 
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>


--===============4617478550770594663==--