Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Thu, 16 Jul 2015 04:52:05 +0000 (UTC)
From: "Xuefu Zhang (JIRA)" <jira@apache.org>
To: dev@hive.apache.org
Message-ID: <JIRA.12845484.1437022316000.201903.1437022325701@Atlassian.JIRA>
In-Reply-To: <JIRA.12845484.1437022316000@Atlassian.JIRA>
References: <JIRA.12845484.1437022316000@Atlassian.JIRA>
 <JIRA.12845484.1437022316027@arcas>
Subject: [jira] [Created] (HIVE-11276) Optimization around job submission
 and adding jars [Spark Branch]
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Xuefu Zhang created HIVE-11276:
----------------------------------

             Summary: Optimization around job submission and adding jars [Spark Branch]
                 Key: HIVE-11276
                 URL: https://issues.apache.org/jira/browse/HIVE-11276
             Project: Hive
          Issue Type: Sub-task
          Components: Spark
    Affects Versions: 1.1.0
            Reporter: Xuefu Zhang


It seems that Hive on Spark has some room for performance improvement on job submission. Specifically, we are calling refreshLocalResources() for every job submission despite there is are no changes in the jar list. Since Hive on Spark is reusing the containers in the whole user session, we might be able to optimize that.

We do need to take into consideration the case of dynamic allocation, in which new executors might be added.

This task is some R&D in this area.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)