hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Harris <>
Subject RE: make best use of VCore in Hive
Date Tue, 29 Mar 2016 00:18:07 GMT
In my opinion, this ultimately becomes a resource balance issue that you'll need to test.

You have a fixed amount of memory (although you haven't said what it is). As you increase
the number of tasks, the available memory per task will decrease.  If the tasks run out of
memory, they will either page to disk or fail.

If they fail you'll be forced to increase available memory until you figure out how much memory
each task needs to succeed...that will become the limiting factor on how many tasks you can

Hive is generally good about falling back and not failing, but performance may suffer.

But given the situation you describe, I'd generally recommend increasing the number of tasks
and monitoring to ensure that actually improves the performance.

-----Original Message-----
From: mahender bigdata [] 
Sent: Monday, March 28, 2016 6:04 PM
Subject: make best use of VCore in Hive


Currently we are doing join 2-3 big tables and couple of Left Joins. We 
are running on 40 node cluster, During query execution, we could see all 
the memory has been utilized completely (100%), which is perfect. But 
Number of VCore used are less than 50%. Is there a way to increase usage 
of number of cores to almost 90-100% . Can setting Number of Tasks for 
mapper and reduce with TEZ execution makes effectively VCore ?. Any 
guidance on VCore maximum utilization.

information that is privileged and exempt from disclosure under applicable law. If you are
neither the intended recipient nor responsible for delivering the message to the intended
recipient, please note that any dissemination, distribution, copying or the taking of any
action in reliance upon the message is strictly prohibited. If you have received this communication
in error, please notify the sender immediately.  Thank you.
View raw message