hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Shelukhin <ser...@hortonworks.com>
Subject Re: Review Request 48159: HIVE-13901: Hivemetastore add partitions can be slow depending on filesystems
Date Fri, 03 Jun 2016 01:52:33 GMT


> On June 3, 2016, 12:42 a.m., Sergey Shelukhin wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 2357
> > <https://reviews.apache.org/r/48159/diff/2/?file=1405580#file1405580line2357>
> >
> >     cleanup code needs to be updated below. As far as I can tell, when this throws
there can be completed operations yet to be added to the set, unless they are decisively terminated
or allowed to finish
> 
> Rajesh Balamohan wrote:
>     Same approach is followed as in earlier case (where it was all or none gets updated).
 Plz let me know if i am missing anything here.

addedPartitions set is used to determine which directories to delete on clean up.
In the patch, the access to addedPartitions itself doesn't appear to be thread safe. If it
was, there's a question of tasks that are somewhere between creating the directory and adding
to the map.
Actually, I don't think addedPartitions.put inside the callable is thread safe.


- Sergey


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48159/#review136018
-----------------------------------------------------------


On June 3, 2016, 1:40 a.m., Rajesh Balamohan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48159/
> -----------------------------------------------------------
> 
> (Updated June 3, 2016, 1:40 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-13901
>     https://issues.apache.org/jira/browse/HIVE-13901
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Depending on FS, creating external tables & adding partitions can be expensive (e.g
msck which adds all partitions).
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9cc8fbe 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 94dd72e 
> 
> Diff: https://reviews.apache.org/r/48159/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message