hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vaibhav Gumashta <vgumas...@hortonworks.com>
Subject Re: Review Request 62228: HIVE-17495: CachedStore: prewarm improvements, refactoring and caching some aggregate stats
Date Thu, 14 Sep 2017 21:03:01 GMT


> On Sept. 12, 2017, 6:18 a.m., Ashutosh Chauhan wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
> > Lines 275 (patched)
> > <https://reviews.apache.org/r/62228/diff/1/?file=1819749#file1819749line283>
> >
> >     Here we are listing all partitions for table and than we immediately aggr stats
for all partitions. Another (better) way is to not retrieve partNames and do a sql query to
aggr stats for partitions by partFilterExpr. Essentially get_aggr_stats_for(dbName, tblName,
partFilterExpr).
> >     Here, partFilterExpr = * 
> >     That will allow many roundtrips to backend DB.

I'll create a follow-up jira for this (near term item) if you're ok with it.


> On Sept. 12, 2017, 6:18 a.m., Ashutosh Chauhan wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
> > Lines 295 (patched)
> > <https://reviews.apache.org/r/62228/diff/1/?file=1819749#file1819749line303>
> >
> >     And here it will be partFilterExpr = partNames not in (defaultPartition)

I'll create a follow-up jira for this (near term item) if you're ok with it.


> On Sept. 12, 2017, 6:18 a.m., Ashutosh Chauhan wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
> > Line 1536 (original), 1570 (patched)
> > <https://reviews.apache.org/r/62228/diff/1/?file=1819749#file1819749line1578>
> >
> >     This if condition will always be true for cachedstore prewarm invocation.
> >     can you please add comments for that.

Modified prewarm to use the mergeColStatsForPartitions call directly which would also avoid
the extra cache lookup we were doing in prewarm.


- Vaibhav


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62228/#review185136
-----------------------------------------------------------


On Sept. 11, 2017, 9:25 p.m., Vaibhav Gumashta wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62228/
> -----------------------------------------------------------
> 
> (Updated Sept. 11, 2017, 9:25 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Thejas Nair.
> 
> 
> Bugs: HIVE-17495
>     https://issues.apache.org/jira/browse/HIVE-17495
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> https://issues.apache.org/jira/browse/HIVE-17495
> 
> 
> Diffs
> -----
> 
>   itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
8d861e4 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java dc1245e

>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java bbe13fd 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 3053dcb 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 71982a0 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 3ba81ce

>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 80b17e0

>   metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BinaryColumnStatsAggregator.java
e6c836b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BooleanColumnStatsAggregator.java
a34bc9f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/ColumnStatsAggregator.java
a52e5e5 
>   metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/ColumnStatsAggregatorFactory.java
dfae708 
>   metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
ee95396 
>   metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DecimalColumnStatsAggregator.java
284c12c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DoubleColumnStatsAggregator.java
bb4a725 
>   metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/LongColumnStatsAggregator.java
5b1145e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
1b29f92 
>   metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
4db203d 
>   metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
fb16cfc 
> 
> 
> Diff: https://reviews.apache.org/r/62228/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message