hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12951) Reduce Spark executor prewarm timeout to 5s
Date Thu, 28 Jan 2016 02:36:39 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120629#comment-15120629
] 

Rui Li commented on HIVE-12951:
-------------------------------

Thanks Xuefu for the clarifications!
I think "expected resources" is something like {{spark.executor.instances}} (not considering
dynamic allocation). These spark configurations are intended for job execution, i.e. pre-warm
before scheduling any tasks. But they won't help deciding the parallelism on hive side. Therefore
what we do here still makes sense.

The patch LGTM. One concern is that it may cause some tests diff. Otherwise +1.

> Reduce Spark executor prewarm timeout to 5s
> -------------------------------------------
>
>                 Key: HIVE-12951
>                 URL: https://issues.apache.org/jira/browse/HIVE-12951
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 1.2.0
>            Reporter: Xuefu Zhang
>            Assignee: Xuefu Zhang
>         Attachments: HIVE-12951.patch
>
>
> Currently it's set to 30s, which tends to be longer than needed. Reduce it to 5s, only
considering jvm startup time. (Eventually, we may want to make this configurable.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message