hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Gates <ga...@yahoo-inc.com>
Subject Re: Compile Pig and Hive queries to LINQ expression trees?
Date Thu, 30 Oct 2008 17:05:50 GMT
Jeff,

If I understand your proposal it is that Hive SQL and Pig Latin would  
both compile into LINQ Expression Trees as their logical plans, but  
continue to have separate backends for executing the queries.  Is  
that correct?

I'm not seeing the benefit there.  I see the benefit of sharing  
logical plans and a merged backend that can execute both Pig Latin  
and Hive SQL.  These benefits would include focusing more developers  
on what are probably very similar issues that we need to address,  
plus allowing both our use communities to choose which language to  
express their programs in without needing to maintain both systems.   
I also see all of the challenges of merging two projects, the fact  
that we have differing data models, etc.

What do you see as the benefits of sharing just the logical plans?

Alan.

On Oct 27, 2008, at 1:16 PM, Jeff Hammerbacher wrote:

> Hey,
>
> There's been some discussion for a while about having a common logical
> plan format for Pig and Hive to pass to their physical plan
> generators. Erik Meijer gave a talk recently on LINQ Expression Trees
> that made me think they would serve as an excellent intermediate data
> structure. You can read more about them here:
> http://msdn.microsoft.com/en-us/library/bb882636.aspx, and you can
> read the proposal for Erik's presentation here [PDF]:
> http://research.microsoft.com/~emeijer/Papers/Cloud%20computing% 
> 20workshop%20proposal%20Draft.pdf.
>
> I was planning to save this discussion for a time when I understood
> the plan structures in both Pig and Hive, but given the discussion
> around Hive future plans going on right now, I figured now's as good a
> time as any to get it started.
>
> Later,
> Jeff


Mime
View raw message