hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <>
Subject [jira] [Commented] (HIVE-9339) Optimize split grouping for CombineHiveInputFormat [Spark Branch]
Date Tue, 13 Jan 2015 05:36:34 GMT


Rui Li commented on HIVE-9339:

Just some thoughts, since we can't combine the splits in AM like tez, we may need our own
input format (aware of the cluster) to do it on hive side.

> Optimize split grouping for CombineHiveInputFormat [Spark Branch]
> -----------------------------------------------------------------
>                 Key: HIVE-9339
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Jimmy Xiang
> It seems that split generation, especially in terms of grouping inputs, needs to be improved.
For this, we may need cluster information. Because of this, we will first try to solve the
problem for Spark.
> As to cluster information, Spark doesn't provide an API (SPARK-5080). However, Spark
doesn't have a listener API, with which Spark driver can get notifications about executor
going up/down, task starting/finishing, etc. With this information, Spark client should be
able to have a view of the current cluster image.
> Spark developers mentioned that the listener can only be created after SparkContext is
started, at which time, some executions may have already started and so the listener will
miss some information. This can be fixed. File a JIRA with Spark project if necessary.

This message was sent by Atlassian JIRA

View raw message