hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-9339) Optimize split grouping for CombineHiveInputFormat [Spark Branch]
Date Mon, 12 Jan 2015 03:11:34 GMT
Xuefu Zhang created HIVE-9339:
---------------------------------

             Summary: Optimize split grouping for CombineHiveInputFormat [Spark Branch]
                 Key: HIVE-9339
                 URL: https://issues.apache.org/jira/browse/HIVE-9339
             Project: Hive
          Issue Type: Sub-task
          Components: Spark
            Reporter: Xuefu Zhang


It seems that split generation, especially in terms of grouping inputs, needs to be improved.
For this, we may need cluster information. Because of this, we will first try to solve the
problem for Spark.

As to cluster information, Spark doesn't provide an API (SPARK-5080). However, Spark doesn't
have a listener API, with which Spark driver can get notifications about executor going up/down,
task starting/finishing, etc. With this information, Spark client should be able to have a
view of the current cluster image.

Spark developers mentioned that the listener can only be created after SparkContext is started,
at which time, some executions may have already started and so the listener will miss some
information. This can be fixed. File a JIRA with Spark project if necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message