hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Pena <sergio.p...@cloudera.com>
Subject Re: Review Request 57503: HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode
Date Tue, 14 Mar 2017 21:35:50 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57503/#review168942
-----------------------------------------------------------




common/src/test/org/apache/hadoop/hive/common/TestFixedSizeCollection.java
Lines 32 (patched)
<https://reviews.apache.org/r/57503/#comment241253>

    Should we use the java naming convention for method names?



ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java
Line 219 (original), 223-228 (patched)
<https://reviews.apache.org/r/57503/#comment241252>

    Does this mean that if a user attempts to repair a table with more than 500_000 partitions,
the MSCK will fail?
    
    If so, I think we're having the same problem as before. Users won't be able to discover
all partitions from a table with more than 500_000.
    
    Did we have problems before the patch which added this regression issue? If not, should
we use PartitionIterable with unlimited number of partitions instead? The number of HMS transactions
due to PartitionIterable shouldn't be a problem if the user increases the batch size. Also,
as Vihang mentioned, we're just storing two values (partition name + table name), so that
consumes less memory than using the hive.getPartition() method call.


- Sergio Pena


On March 10, 2017, 10:36 a.m., Barna Zsombor Klara wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57503/
> -----------------------------------------------------------
> 
> (Updated March 10, 2017, 10:36 a.m.)
> 
> 
> Review request for hive, Peter Vary, Sergio Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-16024: MSCK Repair Requires nonstrict hive.mapred.mode
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/common/FixedSizeCollection.java PRE-CREATION

>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a479deb7c0c6b779277f1029009b7dfab6dcb9e3

>   common/src/test/org/apache/hadoop/hive/common/TestFixedSizeCollection.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 6805c17a116f5ef0febd36c59d454fa631ae0024

>   ql/src/test/queries/clientnegative/msck_repair_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_0.q ce8ef426a2a58845afc8333259d66725db416584

>   ql/src/test/results/clientnegative/msck_repair_4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_0.q.out 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba

> 
> 
> Diff: https://reviews.apache.org/r/57503/diff/1/
> 
> 
> Testing
> -------
> 
> Tested locally and added qtests/unit tests.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message