Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6997C10211 for ; Tue, 18 Feb 2014 19:06:28 +0000 (UTC) Received: (qmail 43790 invoked by uid 500); 18 Feb 2014 19:06:26 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 43724 invoked by uid 500); 18 Feb 2014 19:06:26 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 43714 invoked by uid 99); 18 Feb 2014 19:06:26 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Feb 2014 19:06:26 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id ECD5F1C00F6; Tue, 18 Feb 2014 19:06:25 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============5995278387233491049==" MIME-Version: 1.0 Subject: Re: Review Request 18200: HIVE-860 - Persistent distributed cache From: "Brock Noland" To: "Xuefu Zhang" , "hive" , "Brock Noland" Date: Tue, 18 Feb 2014 19:06:25 -0000 Message-ID: <20140218190625.4362.85455@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Brock Noland" X-ReviewGroup: hive X-ReviewRequest-URL: https://reviews.apache.org/r/18200/ X-Sender: "Brock Noland" References: <20140218190259.4367.17332@reviews.apache.org> In-Reply-To: <20140218190259.4367.17332@reviews.apache.org> Reply-To: "Brock Noland" X-ReviewRequest-Repository: hive-git --===============5995278387233491049== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit > On Feb. 18, 2014, 7:02 p.m., Xuefu Zhang wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/JarCache.java, line 91 > > > > > > cool. But what about the old cached files? Does HDFS clean them up automatically? The files will have to be periodically cleaned up by the user or admin. Often times admins have policies that delete files which have not been accessed for a long time like a year. We update the access time on files once per day for this purpose. In reality most users won't be using thousands of jars so they won't have to clean them up. - Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18200/#review34740 ----------------------------------------------------------- On Feb. 18, 2014, 3:36 a.m., Brock Noland wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18200/ > ----------------------------------------------------------- > > (Updated Feb. 18, 2014, 3:36 a.m.) > > > Review request for hive. > > > Bugs: HIVE-860 > https://issues.apache.org/jira/browse/HIVE-860 > > > Repository: hive-git > > > Description > ------- > > Caches auxiliary jars and remote runtime jars in /user/$user/.hiveJars by their sha1 hash. This results in: > > 1) faster queries > 2) less distributed cache churn > 3) a smaller/cleaner hive-exec jar > > > Diffs > ----- > > bin/hive 3bd949f > packaging/src/main/assembly/bin.xml a97ef7d > ql/pom.xml 53d0b9e > ql/src/java/org/apache/hadoop/hive/ql/exec/HiveAuxClasspathBuilder.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 288da8e > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/JarCache.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 326654f > shims/aggregator/pom.xml 7aa8c4c > > Diff: https://reviews.apache.org/r/18200/diff/ > > > Testing > ------- > > Tested manually on a cluster. > > > Thanks, > > Brock Noland > > --===============5995278387233491049==--