hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <>
Subject [jira] [Commented] (HIVE-12951) Reduce Spark executor prewarm timeout to 5s
Date Thu, 28 Jan 2016 05:11:39 GMT


Xuefu Zhang commented on HIVE-12951:

Hi [~lirui], that's an interesting idea. I can see that prewarming executors is no longer
needed if parallelism is determined by the expected executors. However, prewarming is meant
for shor-lived sessions, such as ETL job launched by Oozie. For interactive sessions, this
feature doesn't matter much. Therefore, we probably don't want to change the way of determining
parallelism just for a less common use case.

Now let's see if using expected number of executors to determine makes sense in the general
case. In this case, we don't worry about the performance for the first query. Whether we have
static or dynamic allocation, the parallelism is determined in the same way if we use the
actual number of executors. On the other hand, if we use the expected number of executors,
it seems the expected number of executors is undefined in case of dynamic allocation. Also,
the expected may not always match the actual. If there is a big difference, there could be
performance implications.

In conclusion, I can certainly see some benefits of using the expected number of executors.
However, there also seem some unknowns as well. I think we don't have to make a call at this
point. Maybe we can revisit once we learn more or get more user feedback?

What do you think? 

> Reduce Spark executor prewarm timeout to 5s
> -------------------------------------------
>                 Key: HIVE-12951
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 1.2.0
>            Reporter: Xuefu Zhang
>            Assignee: Xuefu Zhang
>         Attachments: HIVE-12951.patch
> Currently it's set to 30s, which tends to be longer than needed. Reduce it to 5s, only
considering jvm startup time. (Eventually, we may want to make this configurable.)

This message was sent by Atlassian JIRA

View raw message