hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasad Mujumdar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8424) Support fair scheduler user queue mapping in non-impersonation mode
Date Mon, 13 Oct 2014 00:47:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168867#comment-14168867
] 

Prasad Mujumdar commented on HIVE-8424:
---------------------------------------

To add to Szehon's comment, this patch is addressing the fair scheduler queue mapping for
HiveServer2 running in non-impersonation mode. HiveServer2 can internally handle other processing,
eg authorization correctly since it has the identity of the end user. However when at MR/Yarn
level, all the the jobs are submitted as user hive. Hence the fair scheduler will use user
Hive's queue for all Hive jobs.
To address this issue, Yarn has exposed the queue mapping API for downstream services like
Hive. This patch invokes that API to figure out the correct queue mapping from Yarn. Then
explicitly sets that queue in the job configuration. This ensures that the jobs for the given
use will get mapped to the appropriate queue.
The patch is enabling this queue re-mapping in by default. This can be disabled by setting
that config property to false.

> Support fair scheduler user queue mapping in non-impersonation mode
> -------------------------------------------------------------------
>
>                 Key: HIVE-8424
>                 URL: https://issues.apache.org/jira/browse/HIVE-8424
>             Project: Hive
>          Issue Type: Improvement
>          Components: Shims
>            Reporter: Mohit Sabharwal
>            Assignee: Mohit Sabharwal
>         Attachments: HIVE-8424.1.patch, HIVE-8424.patch
>
>
> Under non-impersonation mode, all MR jobs run as the hive system user. The default scheduler
queue mapping is one queue per user. This is problematic for users who use the queues to regulate
and track their MR resource usage.
> Yarn exposes an API to retrieve the fair scheduler queue mapping, which we can use to
set the appropriate MR queue for the current user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message