hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From j.prasant...@gmail.com
Subject Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in cache can impact heap usage
Date Fri, 08 Sep 2017 18:24:39 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/#review184992
-----------------------------------------------------------




llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 52 (patched)
<https://reviews.apache.org/r/60289/#comment261216>

    This file looks renamed from Parquet to generic MetadataCache but contains OrcSpecific
objects. If it is generic remove orc related stuff or rename the class if it is orc specific?



llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 106 (patched)
<https://reviews.apache.org/r/60289/#comment261217>

    why lock and unlock notification back to back?



llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 134 (patched)
<https://reviews.apache.org/r/60289/#comment261218>

    can you create follow up? will be useful for debugging. or this could be jmx info. something
that can be looked easily instead of logs.



llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 225 (patched)
<https://reviews.apache.org/r/60289/#comment261221>

    smallBuffer is not added LlapMetadataBuffers?



llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 235 (patched)
<https://reviews.apache.org/r/60289/#comment261222>

    is readFully fixed in hadoop 2.8? if so, now that hive moved to 2.8.0 can that be used
here and other places?



llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 244 (patched)
<https://reviews.apache.org/r/60289/#comment261223>

    remove of change to debug/trace log



ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java
Lines 148 (patched)
<https://reviews.apache.org/r/60289/#comment261224>

    May be we should start using TypeDescription everywhere. OrcProto.Type can be huge object
when compared to TypeDescription.


- Prasanth_J


On Sept. 1, 2017, 12:41 a.m., Sergey Shelukhin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60289/
> -----------------------------------------------------------
> 
> (Updated Sept. 1, 2017, 12:41 a.m.)
> 
> 
> Review request for hive, Gopal V and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java c5248ceb5f

>   llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java f42622b892

>   llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
6edd84b8b0 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
b5db3029d1 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
dc053ee7cf 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java b9d7a77d5b

>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
601b622b49 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
4565d11988 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
b61a8ca022 
>   llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
13c7767a3b 
>   llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 03a955c6f7

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 69a9f9f35e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 7540e72b53

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 690cce798e

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java cdd58df370 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java d47ba6b31a 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 77b7f5a2f7 
>   ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out b799527e30 
>   storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 403c3ada61

> 
> 
> Diff: https://reviews.apache.org/r/60289/diff/4/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message