hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zsolt Fekete <fekete.zsolt...@gmail.com>
Subject Re: Review Request 59408: HIVE-16719 HiveMetaTool fails when the data does not fit in memory
Date Fri, 26 May 2017 21:47:07 GMT


> On May 22, 2017, 11:50 a.m., Peter Vary wrote:
> > Thanks for the patch Zsolt!
> > 
> > It is important to be able to check the data even if does not fit into memory! Thanks
for the catch!
> > 
> > As for the patch, few comments from YETUS:
> > - Checkstyle: I will not put all of them here, but there is a problem with the indentation.
Could you please check the checkstyle output of 
> >      
> >      mvn checkstyle:checkstyle -Dcheckstyle.consoleOutput=true -DskipTests -Pitests
-Ptest-patch 
> >      
> > - ASF headers are missing:
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterable.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterator.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/EntityUpdater.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/IDataProvider.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationEntity.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationEntityImplementations.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationUpdater.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/RetrieverIterable.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/ReturnValue.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/UpdateParams.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/java/org/apache/hadoop/hive/metastore/metatool/UriUpdateChecker.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterableTest.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/DataProviderStub.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/EntityUpdaterTest.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/LocationEntityImplementationsTest.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/LocationUpdaterTest.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/ReturnValueTest.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/metatool/UriUpdateCheckerTest.java
> >  !????? /Users/petervary/dev/upstream/hive/metastore/src/test/org/apache/hadoop/hive/metastore/tools/HiveMetaToolTest.java
> >  
> > Thanks,
> > Peter

Thanks Peter, I will fix the checkstyle errors.
I have 2 questions: can you please give some pointers about how can I run Yetus for hive?
And what is "-Ptest-patch"?


- Zsolt


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59408/#review175622
-----------------------------------------------------------


On May 19, 2017, 5:05 p.m., Zsolt Fekete wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59408/
> -----------------------------------------------------------
> 
> (Updated May 19, 2017, 5:05 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Currently HiveMetaTool reads full tables (as DataNucleus entities) into memory by calling
PersistenceManager's retrieveAll().
> 
> See these methods of ObjectStore: updateMDatabaseURI, updateTblPropURI, updateMStorageDescriptorTblPropURI,
updateMStorageDescriptorTblURI, updateSerdeURI.
> 
> This might cause failure when the affected tables (SDS, DBS, TABLE_PARAMS, SD_PARAMS,
SERDES, SERDE_PARAMS) are too big.
> 
> 
> Diffs
> -----
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterable.java
PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterator.java
PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/EntityUpdater.java PRE-CREATION

>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/IDataProvider.java PRE-CREATION

>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationEntity.java PRE-CREATION

>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationEntityImplementations.java
PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/LocationUpdater.java PRE-CREATION

>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/RetrieverIterable.java
PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/ReturnValue.java PRE-CREATION

>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/UpdateParams.java PRE-CREATION

>   metastore/src/java/org/apache/hadoop/hive/metastore/metatool/UriUpdateChecker.java
PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java 22e246f

>   metastore/src/test/org/apache/hadoop/hive/metastore/metatool/BlockRetrieverIterableTest.java
PRE-CREATION 
>   metastore/src/test/org/apache/hadoop/hive/metastore/metatool/DataProviderStub.java
PRE-CREATION 
>   metastore/src/test/org/apache/hadoop/hive/metastore/metatool/EntityUpdaterTest.java
PRE-CREATION 
>   metastore/src/test/org/apache/hadoop/hive/metastore/metatool/LocationEntityImplementationsTest.java
PRE-CREATION 
>   metastore/src/test/org/apache/hadoop/hive/metastore/metatool/LocationUpdaterTest.java
PRE-CREATION 
>   metastore/src/test/org/apache/hadoop/hive/metastore/metatool/ReturnValueTest.java PRE-CREATION

>   metastore/src/test/org/apache/hadoop/hive/metastore/metatool/UriUpdateCheckerTest.java
PRE-CREATION 
>   metastore/src/test/org/apache/hadoop/hive/metastore/tools/HiveMetaToolTest.java PRE-CREATION

> 
> 
> Diff: https://reviews.apache.org/r/59408/diff/2/
> 
> 
> Testing
> -------
> 
> The new tests passed:
> mvn test -Dtest=BlockRetrieverIterableTest,EntityUpdaterTest,UriUpdateCheckerTest,LocationUpdaterTest,ReturnValueTest,LocationEntityImplementationsTest,IntegrationTest
> 
> 
> Thanks,
> 
> Zsolt Fekete
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message