hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal Vijayaraghavan <gop...@apache.org>
Subject Re: Hive on Tez
Date Wed, 10 Jun 2015 07:28:35 GMT
Hi,

There¹s no relationship between number of containers and tasks ­ well the
number of tasks is the maximum number of containers you can use.

You can run an entire vertex containing many task attempts in one container
if there are no more available ­ because of container reuse.

The memory/cpu settings are actually setup via a configuration parameter ­
hive.tez.container.size.

The Vertex is expanded into multiple tasks ­ The number of map-tasks are
determined by the split-grouping
(tez.grouping.min-size/tez.grouping.split-waves) and the reducers are
estimated from the ReduceSink statistics (divided by
hive.exec.bytes.per.reducer).

Even the reducer number is not final, since the plan-time value is only the
max value for that - you can schedule 1009 reducers and end up only running
11, with Tez auto-reducer parallelism, which only merges adjacent reducers.

This is split between the Tez SplitGrouper, HiveSplitGenerator and
SetReducerParallelism.

Cheers,
Gopal

From:  Yunqi Zhang <yunqi@umich.edu>
Reply-To:  "user@hive.apache.org" <user@hive.apache.org>
Date:  Tuesday, June 9, 2015 at 5:07 PM
To:  "user@hive.apache.org" <user@hive.apache.org>
Subject:  Hive on Tez

Hi guys,
 
I¹m playing with the code that integrates Hive on Tez, and have couple
questions regarding to the resource allocation.
 
To my understanding (correct me if I am wrong), Hive creates a DAG composed
of MapVertex and ReduceVertex, where each Vertex will later be translated to
task running on potentially multiple containers by Tez. I was wondering how
the resource requirement is determined (how many containers are needed for
each Vertex, and what are the requirements for CPU and memory, etc.) in the
current implementation, and where I can find the code corresponding to this.
 
Thank you!


Yunqi



Mime
View raw message