hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brock Noland (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8574) Enhance metrics gathering in Spark Client [Spark Branch]
Date Wed, 26 Nov 2014 19:00:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14226651#comment-14226651
] 

Brock Noland commented on HIVE-8574:
------------------------------------

bq. So unless we're worried about a single job creating so many tasks that it will run the
driver out of memory with all the metrics data, this shouldn't really be an issue.

Any idea how much memory would be consumed for say 100K tasks?

> Enhance metrics gathering in Spark Client [Spark Branch]
> --------------------------------------------------------
>
>                 Key: HIVE-8574
>                 URL: https://issues.apache.org/jira/browse/HIVE-8574
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>
> The current implementation of metrics gathering in the Spark client is a little hacky.
First, it's awkward to use (and the implementation is also pretty ugly). Second, it will just
collect metrics indefinitely, so in the long term it turns into a huge memory leak.
> We need a simplified interface and some mechanism for disposing of old metrics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message