Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
 boundary="===============0425817109951616354=="
MIME-Version: 1.0
Subject: Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in
 cache can impact heap usage
From: j.prasanth.j@gmail.com
To: Gopal V <gopal@hortonworks.com>, j.prasanth.j@gmail.com
Cc: Sergey Shelukhin <sergey@hortonworks.com>, hive <dev@hive.apache.org>
Date: Fri, 08 Sep 2017 18:24:39 -0000
Message-ID: <20170908182439.9617.74571@reviews-vm2.apache.org>
Auto-Submitted: auto-generated
Sender: noreply@reviews.apache.org
References: <20170901004122.9700.12763@reviews-vm2.apache.org>
In-Reply-To: <20170901004122.9700.12763@reviews-vm2.apache.org>
Reply-To: j.prasanth.j@gmail.com
archived-at: Fri, 08 Sep 2017 18:24:53 -0000

--===============0425817109951616354==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/#review184992
-----------------------------------------------------------


llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 52 (patched)
<https://reviews.apache.org/r/60289/#comment261216>

    This file looks renamed from Parquet to generic MetadataCache but contains OrcSpecific objects. If it is generic remove orc related stuff or rename the class if it is orc specific?


llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 106 (patched)
<https://reviews.apache.org/r/60289/#comment261217>

    why lock and unlock notification back to back?


llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 134 (patched)
<https://reviews.apache.org/r/60289/#comment261218>

    can you create follow up? will be useful for debugging. or this could be jmx info. something that can be looked easily instead of logs.


llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 225 (patched)
<https://reviews.apache.org/r/60289/#comment261221>

    smallBuffer is not added LlapMetadataBuffers?


llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 235 (patched)
<https://reviews.apache.org/r/60289/#comment261222>

    is readFully fixed in hadoop 2.8? if so, now that hive moved to 2.8.0 can that be used here and other places?


llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 244 (patched)
<https://reviews.apache.org/r/60289/#comment261223>

    remove of change to debug/trace log


ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java
Lines 148 (patched)
<https://reviews.apache.org/r/60289/#comment261224>

    May be we should start using TypeDescription everywhere. OrcProto.Type can be huge object when compared to TypeDescription.


- Prasanth_J


On Sept. 1, 2017, 12:41 a.m., Sergey Shelukhin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60289/
> -----------------------------------------------------------
> 
> (Updated Sept. 1, 2017, 12:41 a.m.)
> 
> 
> Review request for hive, Gopal V and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
>   llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java c5248ceb5f 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java f42622b892 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 6edd84b8b0 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b5db3029d1 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java dc053ee7cf 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java b9d7a77d5b 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 601b622b49 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 4565d11988 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java b61a8ca022 
>   llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 13c7767a3b 
>   llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 03a955c6f7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 69a9f9f35e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 7540e72b53 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 690cce798e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java cdd58df370 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java d47ba6b31a 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19 
>   ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 77b7f5a2f7 
>   ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out b799527e30 
>   storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 403c3ada61 
> 
> 
> Diff: https://reviews.apache.org/r/60289/diff/4/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>


--===============0425817109951616354==--