hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <>
Subject [jira] Commented: (HIVE-1107) Generic parallel execution framework for Hive (and Pig, and ...)
Date Wed, 17 Nov 2010 21:49:16 GMT


Arun C Murthy commented on HIVE-1107:

I think Russel did a good job explaining it. 

I'll add some more:

I believe, mid-to-long term, that Pig/Hive will get significantly smarter about the way they
construct MR jobs - they will want to run some of the nodes in the DAG, wait for their output
(e.g. a sampler) and then make ever more complicated decisions to modify the DAG. I believe
Oozie isn't the right tool to be using for this purpose. Also, there is no need to force Oozie
either, people can use Azkaban etc. for workflow. 

> Generic parallel execution framework for Hive (and Pig, and ...)
> ----------------------------------------------------------------
>                 Key: HIVE-1107
>                 URL:
>             Project: Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Carl Steinbach
> Pig and Hive each have their own libraries for handling plan execution. As we prepare
to invest more time improving Hive's plan execution mechanism we should also start to consider
ways of building a generic plan execution mechanism that is capable of supporting the needs
of Hive and Pig, as well as other Hadoop data flow programming environments. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message