hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: custom Hive artifacts for Shark project
Date Mon, 26 Aug 2013 04:26:05 GMT
Hi Edward,

Shark is using two jar files from Hive - hive-common and hive-cli. But Shark
community puts a few patches on top of the stock Hive to fix blocking issues
in the latter. The changes aren't proprietary and are either backports from
the newer releases or fixes that weren't committed yet (HIVE-3772 is good
example of this).

Taking into example Hive 0.9 which Shark 0.7 uses. Shark backports a few
bugfixes that were committed into Hive 0.10 or Hive 0.11, but never made it
into Hive 0.9. I believe this is a side effect of Hive always moving forward
and (almost) never making maintenance releases.

Changes and especially massive rewrites bring instability into the software.
It needs to be gradually ironed out with consequent releases. A good example
of such a project would be HBase, that does quite a number of minor releases
to provide their users with stable and robust server-side software. In the
absence of maintenance releases downstream projects tend to find ways to work
around such an obstacle. Hence my earlier email.

As of 0.11.1: Shark currently doesn't support Hive 0.11 because of significant
changes in the APIs of the latter. The support is coming in the next a couple
of months. So, publishing artifacts improving on top of Hive 0.9 might be more
a pressing issue.

Hope it clarifies the situation,
  Cos

On Sun, Aug 25, 2013 at 11:54PM, Edward Capriolo wrote:
> I think we plan on doing an 11.1 or just a 12.0. How does shark use hive?
> Do you just include hive components from maven or does the project somehow
> encorportate our build infrastructure.
> 
> 
> On Sun, Aug 25, 2013 at 7:42 PM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> > Guys,
> >
> > considering the absence of the input, I take it that it really doesn't
> > matter
> > which way the custom artifact will be published. Is it a correct
> > impression?
> >
> > My first choice would be
> >     org.apache.hive.hive-common;0.9-shark0.7
> >     org.apache.hive.hive-cli;0.9-shark0.7
> > artifacts.
> > If this meets the objections from the community here, then I'd like to
> > proceed
> > with
> >     org.shark-project.hive-common;0.9.0
> >     org.shark-project.hive-cli;0.9.0
> >
> > Any of the artifacts are better be published at Maven central to make it
> > readily available for development community.
> >
> > Thoughts?
> > Regards,
> >   Cos
> >
> > On Sat, Aug 10, 2013 at 10:08PM, Konstantin Boudnik wrote:
> > > Guys,
> > >
> > > I am trying to help Spark/Shark community (spark-project.org and now
> > > http://incubator.apache.org/projects/spark) with a predicament. Shark -
> > that's
> > > also known as Hive on Spark - is using some parts of Hive, ie HQL parser,
> > > query optimizer, serdes, and codecs.
> > >
> > > In order to improve some known issues with performance and/or concurrency
> > > Shark developers need to apply a couple of patches on top of the stock
> > Hive:
> > >    https://issues.apache.org/jira/browse/HIVE-2891
> > >    https://issues.apache.org/jira/browse/HIVE-3772 (just committed to
> > trunk)
> > > (as per https://github.com/amplab/shark/wiki/Hive-Patches)
> > >
> > > The issue here is that latest Shark is working on top if Hive 0.9 (Hive
> > 0.11
> > > work is underway) and having developers to apply the patches and build
> > > their own version of the Hive is an extra step that can be avoided.
> > >
> > > One way to address it is to publish Shark specific versions of Hive
> > artifacts
> > > that would have all needed patches applied to stock release.  This way
> > > downstream projects can simply reference the version org.apache.hive with
> > > version 0.9.0-shark-0.7 instead of building Hive locally every time.
> > >
> > > Perhaps this approach is a little overkill, so perhaps if Hive community
> > is
> > > willing to consider a maintenance release of Hive 0.9.1 and perhaps
> > 0.11.1
> > > to include fixes needed by Shark project?
> > >
> > > I am willing to step up and produce Hive release bits if any of the
> > committers
> > > here can help with publishing.
> > >
> > > --
> > > Thanks in advance,
> > >       Cos
> > >
> >
> >
> >

Mime
View raw message