hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Shelukhin <ser...@hortonworks.com>
Subject Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in cache can impact heap usage
Date Thu, 29 Jun 2017 21:20:58 GMT


> On June 29, 2017, 6:38 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
> > Lines 614 (patched)
> > <https://reviews.apache.org/r/60289/diff/1/?file=1757647#file1757647line665>
> >
> >     can this all be baked in OrcFileMetadata class? since stats, stripes are derived
from tail, will be useful if we can just store OrcTail in OrcFileMetadata and lazily construct
stats and stripes when getter is invoked.

why? I'm assuming the reader would always need those and they are already in the object


- Sergey


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/#review179152
-----------------------------------------------------------


On June 21, 2017, 8:31 p.m., Sergey Shelukhin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60289/
> -----------------------------------------------------------
> 
> (Updated June 21, 2017, 8:31 p.m.)
> 
> 
> Review request for hive, Gopal V and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java be38f381e6 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java c73f1a1a7d

>   llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 53c9bae5c1

>   llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
2a76f5c4da 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
dc053ee7cf 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java b9d7a77d5b

>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
601b622b49 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
4565d11988 
>   llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
13c7767a3b 
>   llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 03a955c6f7

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ef7c758d4 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 7540e72b53

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java d5807b77e2

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 31b0609b83 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 4856fb3ceb 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out cd7a392e08 
>   ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out b799527e30 
> 
> 
> Diff: https://reviews.apache.org/r/60289/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message