spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dean Wampler <deanwamp...@gmail.com>
Subject Re: How spark and hive integrate in long term?
Date Fri, 21 Nov 2014 23:12:18 GMT
I can't comment on plans for Spark SQL's support for Hive, but several
companies are porting Hive itself onto Spark:

http://blog.cloudera.com/blog/2014/11/apache-hive-on-apache-spark-the-first-demo/

I'm not sure if they are leveraging the old Shark code base or not, but it
appears to be a fresh effort.

dean

Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com

On Fri, Nov 21, 2014 at 2:51 PM, Zhan Zhang <zhazhan@gmail.com> wrote:

> Now Spark and hive integration is a very nice feature. But I am wondering
> what the long term roadmap is for spark integration with hive. Both of
> these
> two projects are undergoing fast improvement and changes. Currently, my
> understanding is that spark hive sql part relies on hive meta store and
> basic parser to operate, and the thrift-server intercept hive query and
> replace it with its own engine.
>
> With every release of hive, there need a significant effort on spark part
> to
> support it.
>
> For the metastore part, we may possibly replace it with hcatalog. But given
> the dependency of other parts on hive, e.g., metastore, thriftserver,
> hcatlog may not be able to help much.
>
> Does anyone have any insight or idea in mind?
>
> Thanks.
>
> Zhan Zhang
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/How-spark-and-hive-integrate-in-long-term-tp9482.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message