hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Pena <sergio.p...@cloudera.com>
Subject Re: [DISCUSS] Making storage-api a separately released artifact
Date Fri, 26 Aug 2016 19:58:52 GMT
Question:

Wouldn't be better to move part of the implementations to Orc, Parquet and
Avro, and just have some interfaces and basic implementations on Hive? This
way we could avoid Orc, Parquet and/or Avro to depend from Hive. I saw this
on Parquet where they created a RowBatch class internally and returns that
to Hive, then in Hive we will just bind it to the Hive vectorized interface
to support vectorization. It justs an idea, I am not clear exactly what I
am trying to say :)


On Fri, Aug 19, 2016 at 11:01 PM, Lefty Leverenz <leftyleverenz@gmail.com>
wrote:

> Sergey's idea is creative, although it leads to confusion about JIRA fix
> versions.  Issues would be given fix versions based on assumptions about
> whether SA or Hive will be released first.  (That's hard to predict when
> it's months away.)
>
> Keeping the version numbers tied together is very appealing.  Would it be
> possible to have incompatible changes in SA force a bump in the Hive
> release number?  Hm, I guess that means Hive would need a release at the
> same time as SA, but only for incompatible changes.
>
> What's the likelihood of another subproject getting spun off eventually?
> If that happened, the 4th minor version wouldn't make sense.  A 5th minor
> version wouldn't work either.
>
> -- Lefty
>
>
> On Fri, Aug 19, 2016 at 9:46 PM, Sergey Shelukhin <sergey@hortonworks.com>
> wrote:
>
> > I am suggesting we always skip the number. So only one component gets the
> > next one :) In your example Hive trunk would be 2.3, and if SA is
> released
> > again it would become 2.4. Otherwise we’d need a compat table cause
> > versions will be totally out of sync.
> >
> > On 16/8/19, 16:31, "Owen O'Malley" <omalley@apache.org> wrote:
> >
> > >That won't necessarily work, especially in the beginning. If we release
> SA
> > >2.2.0 and use it for Hive trunk with the assumption that the next Hive
> > >release will be 2.2. What do we do when we need to make an incompatible
> > >change in SA? I guess we could release SA as 2.3.0 and when hive makes
> its
> > >next release skip over Hive 2.2 and go straight to Hive 2.3.0. In
> general
> > >I
> > >think that we'd be better off with the release numbers not tied
> together.
> > >
> > >.. Owen
> > >
> > >On Fri, Aug 19, 2016 at 4:14 PM, Sergey Shelukhin <
> sergey@hortonworks.com
> > >
> > >wrote:
> > >
> > >> Can we just run the versions thru? I.e. increment it every time but
> > >> release only one component (or both if they happen to align I guess).
> > >> E.g. storage-api will be released at 2.2, and say 2.3 if it moves
> fast,
> > >> then Hive 2.4, then storage-api 2.5, etc.
> > >> That might make it easier to reason about compatibility because the
> > >>order
> > >> is obvious.
> > >>
> > >> On 16/8/19, 09:04, "Sergio Pena" <sergio.pena@cloudera.com> wrote:
> > >>
> > >> >I see Parquet is currently using the SearchArgument class for
> > >>predicates
> > >> >push down.
> > >> >Will this class be part of the new sub-module or project?
> > >> >
> > >> >Following Sushanth idea, can we have other API interfaces in the new
> > >> >project that other components can use?
> > >> >Perhaps having this may be a good reason to create a project.
> > >> >
> > >> >I'm -1 with the 4th minor version. As Owen mentioned, changing the
> 4th
> > >> >version number for incompatible changes is ugly and confusing.
> > >> >I like the new project idea more, +1, but  the storage-api may be too
> > >> >small
> > >> >for a new project.
> > >> >
> > >> >- Sergio
> > >> >
> > >> >On Wed, Aug 17, 2016 at 2:05 PM, Owen O'Malley <omalley@apache.org>
> > >> wrote:
> > >> >
> > >> >> On Wed, Aug 17, 2016 at 10:46 AM, Alan Gates <alanfgates@gmail.com
> >
> > >> >>wrote:
> > >> >>
> > >> >> > +1 for making the API clean and easy for other projects to
work
> > >>with.
> > >> >> A
> > >> >> > few questions:
> > >> >> >
> > >> >> > 1) Would this also make it easier for Parquet and others
to
> > >>implement
> > >> >> > Hive’s ACID interfaces?
> > >> >> >
> > >> >>
> > >> >> Currently the ACID interfaces haven't been moved over to
> storage-api,
> > >> >> although it would make sense to do so at some point.
> > >> >>
> > >> >>
> > >> >> >
> > >> >> > 2) Would we make any attempt to coordinate version numbers
> between
> > >> >>Hive
> > >> >> > and the storage module, or would a given version of Hive
just
> > >>depend
> > >> >>on a
> > >> >> > given version of the storage module?
> > >> >> >
> > >> >>
> > >> >> The two options that I see are:
> > >> >>
> > >> >> * Let the numbers run separately starting from 2.2.0.
> > >> >> * Tie the numbers together with an additional level of versioning
> > >>(eg.
> > >> >> 2.2.0.0).
> > >> >>
> > >> >> I think that letting the two version numbers diverge is better
in
> the
> > >> >>long
> > >> >> term. For example, if you need to make an incompatible change,
it
> is
> > >> >>pretty
> > >> >> ugly to do it as a fourth level version number (eg. an incompatible
> > >> >>change
> > >> >> from 2.2.0.0 to 2.2.0.1). At the beginning, I expect that
> storage-api
> > >> >>would
> > >> >> move faster than Hive, but as it stabilizes I expect it might
start
> > >> >>moving
> > >> >> slower than Hive.
> > >> >>
> > >> >> I'd propose that we have Hive's build use a released version of
> > >> >>storage-api
> > >> >> rather than a snapshot.
> > >> >>
> > >> >> Thoughts?
> > >> >>
> > >> >>    Owen
> > >> >>
> > >> >>
> > >> >> > Alan.
> > >> >> >
> > >> >> > > On Aug 15, 2016, at 17:01, Owen O'Malley <omalley@apache.org>
> > >> wrote:
> > >> >> > >
> > >> >> > > All,
> > >> >> > >
> > >> >> > > As part of moving ORC out of Hive, we pulled all of
the
> > >> >>vectorization
> > >> >> > > storage and sarg classes into a separate module, which
is named
> > >> >> > > storage-api.  Although it is currently only used by
ORC, it
> > >>could be
> > >> >> used
> > >> >> > > by Parquet or Avro if they wanted to make a fast vectorized
> > >>reader
> > >> >>that
> > >> >> > > read directly in to Hive's VectorizedRowBatch without
needing a
> > >> >>shim or
> > >> >> > > data copy. Note that this is in many ways similar to
pulling
> the
> > >> >>Arrow
> > >> >> > > project out of Drill.
> > >> >> > >
> > >> >> > > This unfortunately still leaves us with a circular dependency
> > >> >>between
> > >> >> > Hive
> > >> >> > > and ORC. I'd hoped that storage-api wouldn't change
that much,
> > >>but
> > >> >>that
> > >> >> > > doesn't seem to be happening. As a result, ORC ends
up shipping
> > >>its
> > >> >>own
> > >> >> > > fork of storage-api.
> > >> >> > >
> > >> >> > > Although we could make a new project for just the storage-api,
> I
> > >> >>think
> > >> >> it
> > >> >> > > would be better to make it a subproject of Hive that
is
> released
> > >> >> > > independently.
> > >> >> > >
> > >> >> > > What do others think?
> > >> >> > >
> > >> >> > >   Owen
> > >> >> >
> > >> >> >
> > >> >>
> > >>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message