hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11526) LLAP: implement LLAP UI as a separate service
Date Fri, 14 Aug 2015 02:33:46 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696354#comment-14696354
] 

Sergey Shelukhin commented on HIVE-11526:
-----------------------------------------

We can start with building "realtime" view of LLAP state. Because there's no central AM for
LLAP cluster, it's good to build a separate service. You can see LlapDaemon class for an example
of how to create a daemon. LlapServiceDriver builds the spec for slider to run LLAP cluster,
so this service should be added to that. It probably doesn't need a lot of memory or other
resources.
LLAP exposes JMX view via HTTP, on port 15002 by default (it's in LlapConfiguration file),
I think it should be in the doc I've sent. So for starters this new service could just locate
the LLAP daemons (see for example how Yarn registry used to do this in Tez AM - LlapTaskSchedulerService
has registry and activeInstances fields that it uses to keep track of them), and then periodically
scrape the JMX view (it's JSON iirc), and produce some view.
You can see what useful counters and data this view already has (such as CPU/memory/etc. info,
cache metrics, executor state).
It should then expose a view of its own. 
For now, it can just produce a JMX text view similar to what LLAP daemon produces, but with
aggregated and summarized information. You can see how YARN web app is created for that in
LlapWebServices file and related files; to get JMX output, see interfaces marked with MXBean
 annotation, like LlapDaemonMXBean, and calls to MBeans.register. Whatever is registered will
be output into jmx view by the web app.

This is probably better if broken down into multiple subtasks (create daemon, run it, scrape
data, output data, maybe others), feel free to open them.

This would be a good start.
Then if we have this standing, we can think about having UI, some more advanced views, what
info to add to JMX, etc.

> LLAP: implement LLAP UI as a separate service
> ---------------------------------------------
>
>                 Key: HIVE-11526
>                 URL: https://issues.apache.org/jira/browse/HIVE-11526
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergey Shelukhin
>            Assignee: Kai Sasaki
>
> The specifics are vague at this point. 
> Hadoop metrics can be output, as well as metrics we collect and output in jmx, as well
as those we collect per fragment and log right now. 
> This service can do LLAP-specific views, and per-query aggregation.
> [~gopalv] may have some information on how to reuse existing solutions for part of the
work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message