Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A0A78200AE4 for ; Fri, 10 Jun 2016 02:24:38 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 9F11E160A58; Fri, 10 Jun 2016 00:24:38 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E70A4160A59 for ; Fri, 10 Jun 2016 02:24:37 +0200 (CEST) Received: (qmail 95456 invoked by uid 500); 10 Jun 2016 00:24:36 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 94762 invoked by uid 99); 10 Jun 2016 00:24:36 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2016 00:24:36 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 914822C8113; Fri, 10 Jun 2016 00:24:34 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============1609579625395222744==" MIME-Version: 1.0 Subject: Re: Review Request 48520: Use multi-threaded approach to listing files for msck From: Hari Sankar Sivarama Subramaniyan To: Ashutosh Chauhan Cc: Hari Sankar Sivarama Subramaniyan , pengcheng xiong , hive Date: Fri, 10 Jun 2016 00:24:34 -0000 Message-ID: <20160610002434.8185.90039@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Hari Sankar Sivarama Subramaniyan X-ReviewGroup: hive X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/48520/ X-Sender: Hari Sankar Sivarama Subramaniyan References: <20160609231642.8186.83460@reviews.apache.org> In-Reply-To: <20160609231642.8186.83460@reviews.apache.org> Reply-To: Hari Sankar Sivarama Subramaniyan X-ReviewRequest-Repository: hive-git archived-at: Fri, 10 Jun 2016 00:24:38 -0000 --===============1609579625395222744== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48520/#review136931 ----------------------------------------------------------- ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java (line 379) nit: is it possible to make allDirs as SynchronizedSet so that someone doesnt misuse this in future. ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java (line 385) Can you please update this parameter description in HiveConf. ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java (line 390) nit: Fine to use a Void return type and return null object instead of true always. ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java (line 394) This will be effectively a serial operation if we have a skewed directory structure (very rare or possibly no scenarios ??) Another thing I remembered is that HIVE_MOVE_FILES_THREAD_COUNT does support a value of 0, which runs the entire thing in serial mode. So if you are reusing that configuration, you will have to keep the serial code path or else you need to introduce a new param. Otherwise there will be a conflict. - Hari Sankar Sivarama Subramaniyan On June 9, 2016, 11:16 p.m., pengcheng xiong wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48520/ > ----------------------------------------------------------- > > (Updated June 9, 2016, 11:16 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Repository: hive-git > > > Description > ------- > > HIVE-13984 > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 10fa561 > > Diff: https://reviews.apache.org/r/48520/diff/ > > > Testing > ------- > > > Thanks, > > pengcheng xiong > > --===============1609579625395222744==--