incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Crawford <traviscrawf...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 18:35:53 GMT
Sounds good - fixes for the RC issues identified thus far have been
reviewed. I'll get them into trunk/branch-0.5 and post a new RC.

--travis


On Fri, Jan 25, 2013 at 10:31 AM, Alan Gates <gates@hortonworks.com> wrote:

>
> On Jan 25, 2013, at 9:18 AM, Travis Crawford wrote:
>
> > On Thu, Jan 24, 2013 at 7:36 PM, Sushanth Sowmyan <khorgath@gmail.com>
> wrote:
> > Hmm, looks like the pom.xml used in that bigtop build is different
> > from the one we have in our incubating candidate (probably older?) -
> > it was bringing in hadoop-core, hadoop-test and hadoop-tools as
> > dependencies from the root pom.xml with version ${hadoop20.version}.
> >
> > However, I do still see that webhcat-svr has a dependency on
> > hadoop-core-${hadoop20.version} in the incubating candidate. We may
> > have to wrap that inside a profile.
> >
> > This is intentional because webhcat does not yet build with hadoop 23. I
> believe there are a couple other submodules with this issue as well.
> >
> > Do we see this as a blocker for the 0.5 release? I don't believe full
> hadoop 23 support is expected yet, and I'd prefer not to block the release
> on this. I totally agree full 23 support is important, but think we should
> save it for a future release.
>
> +1.  We have some great new functionality.  Let's get it out there.
>
> Alan.
>
> >
> > Thoughts?
> >
> > --travis
> >
> >
> >
> >
> > On Thu, Jan 24, 2013 at 7:09 PM, Travis Crawford
> > <traviscrawford@gmail.com> wrote:
> > > Huh that's very interesting. If you do a verbose build maven will
> print out
> > > a dep tree and we can see why it's pulled in. Can you reproduce?
> > >
> > > --Travis
> > >
> > >
> > > On Thursday, January 24, 2013, Sushanth Sowmyan wrote:
> > >>
> > >> And also, in this case, in the core/build/lib/compile/ directory, I
> > >> see the following versions of hadoop jars (notice hadoop-core-1.0.3 in
> > >> the midst of the others):
> > >>
> > >> --
> > >> hadoop-annotations-2.0.2.22-alpha.jar
> > >> hadoop-archives-2.0.2.22-alpha.jar
> > >> hadoop-auth-2.0.2.22-alpha.jar
> > >> hadoop-common-2.0.2.22-alpha.jar
> > >> hadoop-core-1.0.3.jar
> > >> hadoop-hdfs-2.0.2.22-alpha.jar
> > >> hadoop-mapreduce-client-core-2.0.2.22-alpha.jar
> > >> hadoop-yarn-api-2.0.2.22-alpha.jar
> > >> hadoop-yarn-common-2.0.2.22-alpha.jar
> > >> --
> > >>
> > >> On Thu, Jan 24, 2013 at 5:03 PM, Sushanth Sowmyan <khorgath@gmail.com
> >
> > >> wrote:
> > >> > Interestingly, I'm hitting the same issue with a bigtop build for
> > >> > hadoop-2. And this one isn't solved by doing a clean.
> > >> >
> > >> > On Thu, Jan 24, 2013 at 4:26 PM, Travis Crawford
> > >> > <traviscrawford@gmail.com> wrote:
> > >> >> On Thu, Jan 24, 2013 at 4:20 PM, Timothy Potter <
> thelabdude@gmail.com>
> > >> >> wrote:
> > >> >>>
> > >> >>> Hi Travis,
> > >> >>>
> > >> >>> Thank you! Using your ant command, I have a build.
> > >> >>
> > >> >>
> > >> >> Do you know what changed? You might have had 0.20 hadoop jars
> already
> > >> >> resolved, which would cause a build failure without cleaning.
> > >> >>
> > >> >>
> > >> >>>
> > >> >>> Are there any instructions or known pitfalls when upgrading
from
> 0.4
> > >> >>> to
> > >> >>> 0.5?
> > >> >>>
> > >> >>> Or, even better - can I just replace the 0.4 JARS in
> > >> >>> /usr/lib/hive/share/hcatalog with the new 0.5 JARs and restart
the
> > >> >>> metastore?
> > >> >>
> > >> >>
> > >> >> The upgrade procedure will differ based on how you have things
> > >> >> deployed,
> > >> >> primarily if you run the HiveMetaStore as a service or you directly
> > >> >> connect
> > >> >> to the HMS. You might try running a job with just the HCatalog
0.5
> > >> >> client
> > >> >> jars against your existing metastore and see if that works for
> you. As
> > >> >> the
> > >> >> vast majority of HCatalog is client-side, its possible to run
jobs
> with
> > >> >> different versions while you work out the upgrade procedure for
> your
> > >> >> site.
> > >> >>
> > >> >> --travis
> > >> >>
> > >> >>
> > >> >>
> > >> >>>
> > >> >>> Cheers,
> > >> >>> Tim
> > >> >>>
> > >> >>> On Thu, Jan 24, 2013 at 5:00 PM, Travis Crawford
> > >> >>> <traviscrawford@gmail.com> wrote:
> > >> >>>>
> > >> >>>> Hey Timothy -
> > >> >>>>
> > >> >>>> On a clean master, this command works for me:
> > >> >>>>
> > >> >>>>     ant clean package -Dmvn.hadoop.profile=hadoop23
> > >> >>>>
> > >> >>>> After doing so, we see the following jars were used in
the build:
> > >> >>>>
> > >> >>>> fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
> > >> >>>> core/build/lib/compile/hadoop-annotations-0.23.3.jar
> > >> >>>> core/build/lib/compile/hadoop-archives-0.23.3.jar
> > >> >>>> core/build/lib/compile/hadoop-auth-0.23.3.jar
> > >> >>>> core/build/lib/compile/hadoop-common-0.23.3.jar
> > >> >>>> core/build/lib/compile/hadoop-hdfs-0.23.3.jar
> > >> >>>> core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
> > >> >>>> core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
> > >> >>>> core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
> > >> >>>> fynn-2:hcatalog travis$
> > >> >>>>
> > >> >>>> I'm not familiar with how CDH4 packages things. Let's
see what
> jars
> > >> >>>> you're pulling in first. Also, are you changing this line
in the
> > >> >>>> top-level
> > >> >>>> pom.xml?
> > >> >>>>
> > >> >>>> <hadoop23.version>0.23.3</hadoop23.version>
> > >> >>>>
> > >> >>>> If not, I'm surprised you see issues as you wouldn't even
be
> > >> >>>> compiling
> > >> >>>> against CDH4.
> > >> >>>>
> > >> >>>> --travis
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>> On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter
> > >> >>>> <thelabdude@gmail.com>
> > >> >>>> wrote:
> > >> >>>>>
> > >> >>>>> Would like to upgrade to 0.5 as we're maintaining
0.4 + many of
> the
> > >> >>>>> patches supplied since the 0.4 release.
> > >> >>>>>
> > >> >>>>> However, I need HCatalog to run against CDH4.1.2 which
based on
> this
> > >> >>>>> link
> > >> >>>>>
> > >> >>>>> (
> https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647
> )
> > >> >>>>> is hadoop-2.0.0+5
> >
>
>

Mime
View raw message