hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ning Zhang <nzh...@fb.com>
Subject Re: Review Request: HIVE-2026. Parallelize UpdateInputAccessTimeHook
Date Thu, 03 Mar 2011 19:14:21 GMT
Hi MIS,

Thanks for the contribution! To allow broader audience to review, can you upload your patch
to the JIRA and the review board (I can help you with the review board if it doesn't allow
you to change the request).

A couple of comments before uploading your patch:

1) the 5 sec keepAliveTime seems low. If the # of threads is more than the # of cores, does
it mean the thread will be terminated after 5 secs after it is waiting to get scheduled?

2) do you need to call execService.shutDown() in case of a Throwable is caught?

On Mar 3, 2011, at 10:09 AM, MIS wrote:

Hi, Ning

Just to be clear on what I was suggesting, I have created a patch only for this file.
Please have a look.

Thanks,
MIS.


On Thu, Mar 3, 2011 at 5:50 PM, M IS <misapache@gmail.com<mailto:misapache@gmail.com>>
wrote:
This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/460/

trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/UpdateInputAccessTimeHook.java<https://reviews.apache.org/r/460/diff/1/?file=13550#file13550line82>
(Diff revision 1)

public void run(SessionState sess, Set<ReadEntity> inputs,


        77

      Thread[] threads = new Thread[nThreads];


How about going for a centralized thread pool and submitting the tasks for that pool.
This can have advantages like, we need not have to create threads and we could come to know
of the status of the task submitted through the future object. And use this future to to wait
till the task is finished. We can re factor the code to make UpdateWorker to implement Runnable
instead of extending of Thread.


- M


On March 3rd, 2011, 12:53 a.m., Ning Zhang wrote:

Review request for hive.
By Ning Zhang.

Updated 2011-03-03 00:53:49

Description

define hive.hooks.parallel.degree to control max # of thread to update metastore in parallel.



Diffs

  *   trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (1076459)
  *   trunk/conf/hive-default.xml (1076459)
  *   trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java (1076459)
  *   trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/UpdateInputAccessTimeHook.java (1076459)

View Diff<https://reviews.apache.org/r/460/diff/>


<HIVE-2026_1.patch>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message