hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mithun Radhakrishnan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-17784) Make Tez AM's Queue headroom calculation and nParallel tasks configurable.
Date Thu, 12 Oct 2017 06:41:00 GMT
Mithun Radhakrishnan created HIVE-17784:
-------------------------------------------

             Summary: Make Tez AM's Queue headroom calculation and nParallel tasks configurable.
                 Key: HIVE-17784
                 URL: https://issues.apache.org/jira/browse/HIVE-17784
             Project: Hive
          Issue Type: Bug
          Components: Query Planning, Tez
    Affects Versions: 2.2.0, 3.0.0
            Reporter: Mithun Radhakrishnan
            Assignee: Mithun Radhakrishnan


Here's a couple of customizations we made at Yahoo with Hive Tez AMs:
# When calculating splits, {{HiveSplitGenerator}} takes the entire queue's capacity as available,
and generates splits accordingly. While this greedy algorithm might be acceptable for exclusive
queues, on a shared queue, greedy queries will hold other queries up. The algorithm that calculates
the queue's headroom should be pluggable. The greedy version can be the default.
# {{TEZ_AM_VERTEX_MAX_TASK_CONCURRENCY}} and the AM's heap-size can be tuned separately from
the AM's container size. We found that users who attempt to increase vertex concurrency tend
to forget to bump AM memory/container sizes. It would be handier if those values were derived
from the container size.

I'm combining these into a single patch, for easier review.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message