hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: [Discuss] project chop up
Date Sat, 27 Jul 2013 04:25:50 GMT
Yeah I think the tar target should build the whole project.
On Jul 26, 2013 11:05 PM, "Alan Gates" <gates@hortonworks.com> wrote:

> But I assume they'd still be a part of targets like package, tar, and
> binary?  Making them compile and test separately and explicitly load the
> core Hive jars from maven/ivy seems reasonable.
>
> Alan.
>
> On Jul 26, 2013, at 8:40 PM, Brock Noland wrote:
>
> > Hi,
> >
> > I think thats part of it but I'd like to decouple the downstream projects
> > even further so that the only connection is the dependency on the hive
> jars.
> >
> > Brock
> > On Jul 26, 2013 10:10 PM, "Alan Gates" <gates@hortonworks.com> wrote:
> >
> >> I'm not sure how this is different from what hcat does today.  It needs
> >> Hive's jars to compile, so it's one of the last things in the compile
> step.
> >> Would moving the other modules you note to be in the same category be
> >> enough?  Did you want to also make it so that the default ant target
> >> doesn't compile those?
> >>
> >> Alan.
> >>
> >> On Jul 26, 2013, at 4:09 PM, Edward Capriolo wrote:
> >>
> >>> My mistake on saying hcat was a fork metastore. I had a brain fart for
> a
> >>> moment.
> >>>
> >>> One way we could do this is create a folder called downstream. In our
> >>> release step we can execute the downstream builds and then copy the
> files
> >>> we need back. So nothing downstream will be on the classpath of the
> main
> >>> project.
> >>>
> >>> This could help us breakup ql as well. Things like exotic file formats
> ,
> >>> and things that are pluggable like zk locking can go here. That might
> be
> >>> overkill.
> >>>
> >>> For now we can focus on building downstream and hivethrift1might be the
> >>> first thing to try to downstream.
> >>>
> >>>
> >>> On Friday, July 26, 2013, Thejas Nair <thejas@hortonworks.com> wrote:
> >>>> +1 to the idea of making the build of core hive and other downstream
> >>>> components independent.
> >>>>
> >>>> bq.  I was under the impression that Hcat and hive-metastore was
> >>>> supposed to merge up somehow.
> >>>>
> >>>> The metastore code was never forked. Hcat was just using
> >>>> hive-metastore and making the metadata available to rest of hadoop
> >>>> (pig, java MR..).
> >>>> A lot of the changes that were driven by hcat goals were being made
in
> >>>> hive-metastore. You can think of hcat as set of libraries that let pig
> >>>> and java MR use hive metastore. Since hcat is closely tied to
> >>>> hive-metastore, it makes sense to have them in same project.
> >>>>
> >>>>
> >>>> On Fri, Jul 26, 2013 at 6:33 AM, Edward Capriolo <
> edlinuxguru@gmail.com
> >>>
> >>> wrote:
> >>>>> Also i believe hcatalog web can fall into the same designation.
> >>>>>
> >>>>> Question , hcatalog was initily a big hive-metastore fork. I was
> under
> >>> the
> >>>>> impression that Hcat and hive-metastore was supposed to merge up
> >> somehow.
> >>>>> What is the status on that? I remember that was one of the core
> reasons
> >>> we
> >>>>> brought it in.
> >>>>>
> >>>>> On Friday, July 26, 2013, Edward Capriolo <edlinuxguru@gmail.com>
> >> wrote:
> >>>>>> I prefer option 3 as well.
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Jul 26, 2013 at 12:52 AM, Brock Noland <brock@cloudera.com>
> >>> wrote:
> >>>>>>>
> >>>>>>> On Thu, Jul 25, 2013 at 9:48 PM, Edward Capriolo <
> >> edlinuxguru@gmail.com
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> I have been developing my laptop on a duel core 2 GB
Ram laptop
> for
> >>>>> years
> >>>>>>>> now. With the addition of hcatalog, hive-thrift2, and
some other
> >>> growth
> >>>>>>>> trying to develop hive in a eclipse on this machine
craws,
> >> especially
> >>>>> if
> >>>>>>>> 'build automatically' is turned on. As we look to add
on more
> things
> >>>>> this
> >>>>>>>> is only going to get worse.
> >>>>>>>>
> >>>>>>>> I am also noticing issues like this:
> >>>>>>>>
> >>>>>>>> https://issues.apache.org/jira/browse/HIVE-4849
> >>>>>>>>
> >>>>>>>> What I think we should do is strip down/out optional
parts of
> hive.
> >>>>>>>>
> >>>>>>>> 1) Hive Hbase
> >>>>>>>> This should really be it's own project to do this right
we really
> >>>>> have to
> >>>>>>>> have multiple branches since hbase is not backwards
compatible.
> >>>>>>>>
> >>>>>>>> 2) Hive Web Interface
> >>>>>>>> Now really a big project but not really critical can
be just as
> >>> easily
> >>>>> be
> >>>>>>>> build separately
> >>>>>>>>
> >>>>>>>> 3) hive thrift 1
> >>>>>>>> We have hive thrift 2 now, it is time for the sun to
set on
> >>>>> hivethrift1,
> >>>>>>>>
> >>>>>>>> 4) odbc
> >>>>>>>> Not entirely convinced about this one but it is really
not
> critical
> >>> to
> >>>>>>>> running hive.
> >>>>>>>>
> >>>>>>>> What I think we should do is create sub-projects for
the above
> >> things
> >>>>> or
> >>>>>>>> simply move them into directories that do not build
with hive.
> >>> Ideally
> >>>>> they
> >>>>>>>> would use maven to pull dependencies.
> >>>>>>>>
> >>>>>>>> What does everyone think?
> >>>>>>>>
> >>>>>>>
> >>>>>>> I agree that projects like the HBase handler and probably
others as
> >>> well
> >>>>>>> should somehow be "downstream" projects which simply depend
on the
> >> hive
> >>>>>>> jars.  I see a couple alternatives for this:
> >>>>>>>
> >>>>>>> * Take the "module" in question to the Apache Incubator
> >>>>>>> * Move the "module" in question to the Apache Extras
> >>>>>>> * Breakup the projects within our own source tree
> >>>>>>>
> >>>>>>> I'd prefer the third option at this point.
> >>>>>>>
> >>>>>>> Brock
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Brock
> >>>>>>
> >>>>>>
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message