hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <>
Subject [jira] [Commented] (HIVE-8118) SparkMapRecorderHandler and SparkReduceRecordHandler should be initialized with multiple result collectors[Spark Branch]
Date Tue, 16 Sep 2014 14:28:34 GMT


Xuefu Zhang commented on HIVE-8118:

Hi [~chengxiang li],

Thank you for your input. I'm not sure if I understand your thought right. Let me clarify
the problem  by giving a SparkWork like this:
MapWork1 -> ReduceWork1
                 \-> ReduceWork2
it means that MapWork1 will generate different datasets to feed to ReduceWork1 and ReduceWork2.
In case of multi-insert, ReduceWork1 and ReduceWork2 will have a FS operator. Inside MapWork1,
there will be two operator branches consuming the same data, and push different data sets
to two RS operators. (ReduceWork1 and ReduceWork2 have different HiveReduceFunctions.)

However, current implemenation only takes the first data set and feed it to both reduce works.
The same problem can happen also if MapWork1 were a reduce work following other ReduceWork
or MapWork.

With this problem, I'm not sure how we can get around without letting MapWork1 generate two
output RDDs, one for each following reduce work. Potentially, we can duplicate MapWork1 and
have the following diagram:
MapWork11 -> ReduceWork1
MapWork12 -> ReduceWork2
where MapWork11 and MapWork12 consume the same input table (input table as RDD), and feed
its first output RDD to ReduceWork1 and the second to ReduceWork2. This has its complexity,
but more importantly, there will be wasted READ (unless SPark is smart enough to cache the
input table, which is unlikely) and COMPUTATION (computing data twice). I feel that it's unlikely
to get such optimizations from Spark framework in the near term.

Thus, I think we have to take into consideration that a map work or a reduce work might generate
multiple RDDs, one feeds to each of its children. Since SparkMapRecorderHandler and SparkReduceRecordHandler
are doing the data processing on map and reduce side, they need to have a way to generate
multiple outputs.

Please correct me if I understood you wrong. Thanks.

> SparkMapRecorderHandler and SparkReduceRecordHandler should be initialized with multiple
result collectors[Spark Branch]
> ------------------------------------------------------------------------------------------------------------------------
>                 Key: HIVE-8118
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Venki Korukanti
>              Labels: Spark-M1
> In the current implementation, both SparkMapRecordHandler and SparkReduceRecorderHandler
takes only one result collector, which limits that the corresponding map or reduce task can
have only one child. It's very comment in multi-insert queries where a map/reduce task has
more than one children. A query like the following has two map tasks as parents:
> {code}
> select name, sum(value) from dec group by name union all select name, value from dec
order by name
> {code}
> It's possible in the future an optimation may be implemented so that a map work is followed
by two reduce works and then connected to a union work.
> Thus, we should take this as a general case. Tez is currently providing a collector for
each child operator in the map-side or reduce side operator tree. We can take Tez as a reference.
> Likely this is a big change and subtasks are possible. 
> With this, we can have a simpler and clean multi-insert implementation. This is also
the problem observed in HIVE-7731.

This message was sent by Atlassian JIRA

View raw message