Return-Path: Delivered-To: apmail-hive-dev-archive@www.apache.org Received: (qmail 58144 invoked from network); 3 Mar 2011 19:14:50 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 3 Mar 2011 19:14:50 -0000 Received: (qmail 47516 invoked by uid 500); 3 Mar 2011 19:14:50 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 47500 invoked by uid 500); 3 Mar 2011 19:14:50 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 47492 invoked by uid 99); 3 Mar 2011 19:14:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 19:14:50 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nzhang@fb.com designates 66.220.144.150 as permitted sender) Received: from [66.220.144.150] (HELO mx-out.facebook.com) (66.220.144.150) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 19:14:44 +0000 Received: from [192.168.18.212] ([192.168.18.212:43394] helo=mail.thefacebook.com) by mta019.snc4.facebook.com (envelope-from ) (ecelerity 2.2.2.45 r(37388)) with ESMTP id BE/9C-04910-F88EF6D4; Thu, 03 Mar 2011 11:14:23 -0800 Received: from SC-MBX02-1.TheFacebook.com ([fe80::e543:4f2a:16b1:b828]) by sc-hub04.TheFacebook.com ([192.168.18.212]) with mapi id 14.01.0270.001; Thu, 3 Mar 2011 11:14:23 -0800 From: Ning Zhang To: MIS CC: hive Subject: Re: Review Request: HIVE-2026. Parallelize UpdateInputAccessTimeHook Thread-Topic: Review Request: HIVE-2026. Parallelize UpdateInputAccessTimeHook Thread-Index: AQHL2YCZ5TC07KmAVEia63SCtNpGtJQcDfuAgABhVACAABIxAA== Date: Thu, 3 Mar 2011 19:14:21 +0000 Message-ID: <5A440139-CBD4-4252-B706-E1A9761E4F94@fb.com> References: <20110303085349.17839.7366@reviews.apache.org> <20110303122053.17839.54325@reviews.apache.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.18.252] Content-Type: multipart/alternative; boundary="_000_5A440139CBD44252B706E1A9761E4F94fbcom_" MIME-Version: 1.0 --_000_5A440139CBD44252B706E1A9761E4F94fbcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi MIS, Thanks for the contribution! To allow broader audience to review, can you u= pload your patch to the JIRA and the review board (I can help you with the = review board if it doesn't allow you to change the request). A couple of comments before uploading your patch: 1) the 5 sec keepAliveTime seems low. If the # of threads is more than the = # of cores, does it mean the thread will be terminated after 5 secs after i= t is waiting to get scheduled? 2) do you need to call execService.shutDown() in case of a Throwable is cau= ght? On Mar 3, 2011, at 10:09 AM, MIS wrote: Hi, Ning Just to be clear on what I was suggesting, I have created a patch only for = this file. Please have a look. Thanks, MIS. On Thu, Mar 3, 2011 at 5:50 PM, M IS > wrote: This is an automatically generated e-mail. To reply, visit: https://reviews= .apache.org/r/460/ trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/UpdateInputAccessTimeHook= .java (Diff revision 1) public void run(SessionState sess, Set inputs, 77 Thread[] threads =3D new Thread[nThreads]; How about going for a centralized thread pool and submitting the tasks for = that pool. This can have advantages like, we need not have to create threads and we co= uld come to know of the status of the task submitted through the future obj= ect. And use this future to to wait till the task is finished. We can re fa= ctor the code to make UpdateWorker to implement Runnable instead of extendi= ng of Thread. - M On March 3rd, 2011, 12:53 a.m., Ning Zhang wrote: Review request for hive. By Ning Zhang. Updated 2011-03-03 00:53:49 Description define hive.hooks.parallel.degree to control max # of thread to update meta= store in parallel. Diffs * trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (1076= 459) * trunk/conf/hive-default.xml (1076459) * trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore= .java (1076459) * trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/UpdateInputAccessTi= meHook.java (1076459) View Diff --_000_5A440139CBD44252B706E1A9761E4F94fbcom_--