hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Kumar <>
Subject Varying vcores/ram for hive queries running Tez engine
Date Mon, 25 Apr 2016 08:27:38 GMT
I was trying to benchmark some hive queries. I am using the tez execution
engine. I varied the values of the following properties:





Changes in values for property 1 is reflected properly. However it seems
that hive does not respect changes in values of property 3; it always
allocates one vcore per requested container (RM is configured to use the
DominantResourceCalculator). This got me thinking about the precedence of
property values in hive and tez.

I have the following questions with respect to these configurations


   Does hive respect the set values for the properties 2 and 3 at all?

   If I set property 1 to a value say 2048 MB and property 2 is set to a
   value of say 1024 MB does this mean that I am wasting about a GB of memory
   for each spawned container?

   Is there a property in hive similar to property 1 that allows me to use
   the 'set' command in the .hql file to specify the number of vcores to use
   per container?

   Changes in value for the property are
   reflected at runtime. However I do not observe the same behaviour with
   property 3. Are there other configurations that take precedence over it?

Your inputs and suggestions would be highly appreciated.


PS: Tests conducted on a 5 node cluster running HDP 2.3.0

View raw message