hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gunther Hagleitner (JIRA)" <>
Subject [jira] [Created] (HIVE-7158) Use Tez auto-parallelism in Hive
Date Sat, 31 May 2014 06:49:01 GMT
Gunther Hagleitner created HIVE-7158:

             Summary: Use Tez auto-parallelism in Hive
                 Key: HIVE-7158
             Project: Hive
          Issue Type: Bug
            Reporter: Gunther Hagleitner
            Assignee: Gunther Hagleitner

Tez can optionally sample data from a fraction of the tasks of a vertex and use that information
to choose the number of downstream tasks for any given scatter gather edge.

Hive estimates the count of reducers by looking at stats and estimates for each operator in
the operator pipeline leading up to the reducer. However, if this estimate turns out to be
too large, Tez can reign in the resources used to compute the reducer.

It does so by combining partitions of the upstream vertex. It cannot, however, add reducers
at this stage.

I'm proposing to let users specify whether they want to use auto-parallelism or not. If they
do there will be scaling factors to determine max and min reducers Tez can choose from. We
will then partition by max reducers, letting Tez sample and reign in the count up until the
specified min.

This message was sent by Atlassian JIRA

View raw message