hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brock Noland" <br...@cloudera.com>
Subject Re: Review Request 18200: HIVE-860 - Persistent distributed cache
Date Tue, 18 Feb 2014 19:06:25 GMT


> On Feb. 18, 2014, 7:02 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/JarCache.java, line 91
> > <https://reviews.apache.org/r/18200/diff/5/?file=494786#file494786line91>
> >
> >     cool. But what about the old cached files? Does HDFS clean them up automatically?

The files will have to be periodically cleaned up by the user or admin. Often times admins
have policies that delete files which have not been accessed for a long time like a year.
We update the access time on files once per day for this purpose. In reality most users won't
be using thousands of jars so they won't have to clean them up.


- Brock


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18200/#review34740
-----------------------------------------------------------


On Feb. 18, 2014, 3:36 a.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18200/
> -----------------------------------------------------------
> 
> (Updated Feb. 18, 2014, 3:36 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-860
>     https://issues.apache.org/jira/browse/HIVE-860
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Caches auxiliary jars and remote runtime jars in /user/$user/.hiveJars by their sha1
hash. This results in:
> 
> 1) faster queries
> 2) less distributed cache churn
> 3) a smaller/cleaner hive-exec jar
> 
> 
> Diffs
> -----
> 
>   bin/hive 3bd949f 
>   packaging/src/main/assembly/bin.xml a97ef7d 
>   ql/pom.xml 53d0b9e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HiveAuxClasspathBuilder.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 288da8e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/JarCache.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 326654f 
>   shims/aggregator/pom.xml 7aa8c4c 
> 
> Diff: https://reviews.apache.org/r/18200/diff/
> 
> 
> Testing
> -------
> 
> Tested manually on a cluster.
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message