hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vihang Karajgaonkar <vih...@cloudera.com>
Subject Re: Review Request 58936: HIVE-16143 : Improve msck repair batching
Date Fri, 05 May 2017 17:23:08 GMT


> On May 5, 2017, 12:54 a.m., Sahil Takiar wrote:
> > common/src/java/org/apache/hive/common/util/RetryUtilities.java
> > Lines 25 (patched)
> > <https://reviews.apache.org/r/58936/diff/1/?file=1706349#file1706349line25>
> >
> >     Might want to looking https://github.com/rholder/guava-retrying
> 
> Sahil Takiar wrote:
>     look into*

Thanks for the pointer. Took a quick look. It has some interesting ideas but it doesn't seem
to support reducing the workload size exponentially. It has a exponential backoff retry interval,
but not the in terms of workload sizing. Also, I have never used this library before. Is it
popular and production ready? The last commit on this library was 2 years ago.


> On May 5, 2017, 12:54 a.m., Sahil Takiar wrote:
> > itests/hive-blobstore/src/test/queries/clientpositive/create_like.q
> > Lines 24 (patched)
> > <https://reviews.apache.org/r/58936/diff/1/?file=1706351#file1706351line24>
> >
> >     is this necessary?

I think this is a better way to the added partitions in the q.out files. The "Repair: Added
partition to metastore..." is added to the output based on the order of iteration over a HashSet
which is not very reliable and prone to flakiness (across different Java distributions and
different versions of same Java).


> On May 5, 2017, 12:54 a.m., Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
> > Lines 1697 (patched)
> > <https://reviews.apache.org/r/58936/diff/1/?file=1706353#file1706353line1697>
> >
> >     Why does this need to be masked?

Same as above. We should not really rely on comparing this string in the q.out file since
the order can change leading to flakiness.


- Vihang


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58936/#review173989
-----------------------------------------------------------


On May 2, 2017, 10:25 p.m., Vihang Karajgaonkar wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58936/
> -----------------------------------------------------------
> 
> (Updated May 2, 2017, 10:25 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Sahil Takiar.
> 
> 
> Bugs: HIVE-16143
>     https://issues.apache.org/jira/browse/HIVE-16143
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-16143 : Improve msck repair batching
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hive/common/util/RetryUtilities.java PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestRetryUtilities.java PRE-CREATION 
>   itests/hive-blobstore/src/test/queries/clientpositive/create_like.q 38f384e4c547d3c93d510b89fccfbc2b8e2cba09

>   itests/hive-blobstore/src/test/results/clientpositive/create_like.q.out 0d362a716291637404a3859fe81068594d82c9e0

>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2ae1eacb68cef6990ae3f2050af0bed7c8e9843f

>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 917e565f28b2c9aaea18033ea3b6b20fa41fcd0a

>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java PRE-CREATION

>   ql/src/test/queries/clientpositive/msck_repair_0.q 22542331621ca4ce5277c2f46a4264b7540a4d1e

>   ql/src/test/queries/clientpositive/msck_repair_1.q ea596cbbd2d4c230f2b5afbe379fc1e8836b6fbd

>   ql/src/test/queries/clientpositive/msck_repair_2.q d8338211e970ebac68a7471ee0960ccf2d51cba3

>   ql/src/test/queries/clientpositive/msck_repair_3.q fdefca121a2de361dbd19e7ef34fb220e1733ed2

>   ql/src/test/queries/clientpositive/msck_repair_batchsize.q e56e97ac36a6544f3e20478fdb0e8fa783a857ef

>   ql/src/test/results/clientpositive/msck_repair_0.q.out 2e0d9dc423071ebbd9a55606f196cf7752e27b1a

>   ql/src/test/results/clientpositive/msck_repair_1.q.out 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba

>   ql/src/test/results/clientpositive/msck_repair_2.q.out 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba

>   ql/src/test/results/clientpositive/msck_repair_3.q.out 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba

>   ql/src/test/results/clientpositive/msck_repair_batchsize.q.out ba99024163a1f2c59d59e9ed7ea276c154c99d24

>   ql/src/test/results/clientpositive/repair.q.out c1834640a35500c521a904a115a718c94546df10

> 
> 
> Diff: https://reviews.apache.org/r/58936/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message