hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Shelukhin <ser...@hortonworks.com>
Subject Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in cache can impact heap usage
Date Thu, 29 Jun 2017 21:19:39 GMT


> On June 29, 2017, 6:38 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java
> > Lines 1685 (patched)
> > <https://reviews.apache.org/r/60289/diff/1/?file=1757656#file1757656line1779>
> >
> >     This looks complicated. It will be better if ORC can provide API that returns
stripe footer and indexes as ByteBuffer which can be directly cached. Stripe footers and Indexes
could be stored with medium priority. 
> >     
> >     Priorities could be:
> >     Serialized file footer - HIGH (this is required to not choke NN, with config
change this is already part of split)
> >     Index + Stripe footer - MEDIUM (with locality re-reading these will not be a
problem)
> >     Data - LOW (same as reading index, stripe footer)
> >     
> >     Since backward seeks no longer close connections for cloud storage, reading
index and stripe could be done faster. 
> >     
> >     I think it would be easier,
> >     if ORC and parquet readers can provide 2 high level interfaces
> >     - Interface to read footers, index as ByteBuffers which LLAP will cache
> >     - Reader interface to accept ByteBuffer from which footers and index can be
read which LLAP or file will provide

I filed some ORC JIRAs to improve APIs... for now a no-op


- Sergey


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/#review179152
-----------------------------------------------------------


On June 21, 2017, 8:31 p.m., Sergey Shelukhin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60289/
> -----------------------------------------------------------
> 
> (Updated June 21, 2017, 8:31 p.m.)
> 
> 
> Review request for hive, Gopal V and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java be38f381e6 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java c73f1a1a7d

>   llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 53c9bae5c1

>   llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
2a76f5c4da 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
dc053ee7cf 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java b9d7a77d5b

>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
601b622b49 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
4565d11988 
>   llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
13c7767a3b 
>   llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 03a955c6f7

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ef7c758d4 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 7540e72b53

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java d5807b77e2

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 31b0609b83 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 4856fb3ceb 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out cd7a392e08 
>   ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out b799527e30 
> 
> 
> Diff: https://reviews.apache.org/r/60289/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message