hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16601) Display Session Id, Query Name / Id, and Dag Id in Spark UI
Date Mon, 16 Oct 2017 15:50:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206080#comment-16206080
] 

Sahil Takiar commented on HIVE-16601:
-------------------------------------

Few updates. The screen shots are a bit outdated, it now looks like:

* The Spark App Name will now be "Hive on Spark (sessionId = \[my-session-id\])"
** Note that users can override the Spark App Name by setting {{spark.app.name}} in their
session config; in this case the "(sessionId = \[my-session-id\])" will be appended to the
Spark App Name
** The goal here is to help users map Hive on Spark applications to the session that launched
them
* The Spark Job List will now list all jobs in a Spark application via the following format:
** "Job Id (Job Group) = \[job-id-number\] (\[Hive-query-id\])" where the Job Id is some monotonically
increasing integer and the Hive-query-id is the query id determined by Hive
*** The Job Group should group Spark jobs from the same Hive query together
** "Description = \[query-being-run\]\[stage-id\]" where the Description is a brief description
of what the Job is doing, the query-being-run is a shortened version of the query being run,
the stage-id is the Hive stage id
*** The description gives more info as to what the Job is actually doing
*** A custom description on the query can be provided by the user via the config {{hive.query.name}}
- in which case the stage-id will automatically be appended to the end of the user specified
query name

> Display Session Id, Query Name / Id, and Dag Id in Spark UI
> -----------------------------------------------------------
>
>                 Key: HIVE-16601
>                 URL: https://issues.apache.org/jira/browse/HIVE-16601
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-16601.1.patch, HIVE-16601.2.patch, HIVE-16601.3.patch, HIVE-16601.4.patch,
HIVE-16601.5.patch, Spark UI Applications List.png, Spark UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the Query Name
/ Id and Dag Id for each Spark job launched. Hive-on-MR does something similar via the {{mapred.job.name}}
parameter. The query name is displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config {{hive.query.name}} for
HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI does something
similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message