hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13376) HoS emits too many logs with application state
Date Wed, 25 May 2016 15:50:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300260#comment-15300260
] 

Rui Li commented on HIVE-13376:
-------------------------------

Spark checks the app state and then (optionally) logs the state report. No job is accepted
before the app reaches RUNNING state. So if Spark waits for 60s before it checks the state,
the first job will have a considerable start-up overhead. You can do some local tests to verify
this.

> HoS emits too many logs with application state
> ----------------------------------------------
>
>                 Key: HIVE-13376
>                 URL: https://issues.apache.org/jira/browse/HIVE-13376
>             Project: Hive
>          Issue Type: Improvement
>          Components: Spark
>            Reporter: Szehon Ho
>            Assignee: Szehon Ho
>             Fix For: 2.1.0
>
>         Attachments: HIVE-13376.2.patch, HIVE-13376.patch
>
>
> The logs get flooded with something like:
> > Mar 28, 3:12:21.851 PM        INFO    org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report for application_1458679386200_0161
(state: RUNNING)
> > Mar 28, 3:12:21.912 PM        INFO    org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:21 INFO yarn.Client: Application report for application_1458679386200_0149
(state: RUNNING)
> > Mar 28, 3:12:22.853 PM        INFO    org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report for application_1458679386200_0161
(state: RUNNING)
> > Mar 28, 3:12:22.913 PM        INFO    org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:22 INFO yarn.Client: Application report for application_1458679386200_0149
(state: RUNNING)
> > Mar 28, 3:12:23.855 PM        INFO    org.apache.hive.spark.client.SparkClientImpl
> > [stderr-redir-1]: 16/03/28 15:12:23 INFO yarn.Client: Application report for application_1458679386200_0161
(state: RUNNING)
> While this is good information, it is a bit much.
> Seems like SparkJobMonitor hard-codes its interval to 1 second.  It should be higher
and perhaps made configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message