incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Crawford <traviscrawf...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 03:09:21 GMT
Huh that's very interesting. If you do a verbose build maven will print out
a dep tree and we can see why it's pulled in. Can you reproduce?

--Travis

On Thursday, January 24, 2013, Sushanth Sowmyan wrote:

> And also, in this case, in the core/build/lib/compile/ directory, I
> see the following versions of hadoop jars (notice hadoop-core-1.0.3 in
> the midst of the others):
>
> --
> hadoop-annotations-2.0.2.22-alpha.jar
> hadoop-archives-2.0.2.22-alpha.jar
> hadoop-auth-2.0.2.22-alpha.jar
> hadoop-common-2.0.2.22-alpha.jar
> hadoop-core-1.0.3.jar
> hadoop-hdfs-2.0.2.22-alpha.jar
> hadoop-mapreduce-client-core-2.0.2.22-alpha.jar
> hadoop-yarn-api-2.0.2.22-alpha.jar
> hadoop-yarn-common-2.0.2.22-alpha.jar
> --
>
> On Thu, Jan 24, 2013 at 5:03 PM, Sushanth Sowmyan <khorgath@gmail.com>
> wrote:
> > Interestingly, I'm hitting the same issue with a bigtop build for
> > hadoop-2. And this one isn't solved by doing a clean.
> >
> > On Thu, Jan 24, 2013 at 4:26 PM, Travis Crawford
> > <traviscrawford@gmail.com> wrote:
> >> On Thu, Jan 24, 2013 at 4:20 PM, Timothy Potter <thelabdude@gmail.com>
> >> wrote:
> >>>
> >>> Hi Travis,
> >>>
> >>> Thank you! Using your ant command, I have a build.
> >>
> >>
> >> Do you know what changed? You might have had 0.20 hadoop jars already
> >> resolved, which would cause a build failure without cleaning.
> >>
> >>
> >>>
> >>> Are there any instructions or known pitfalls when upgrading from 0.4 to
> >>> 0.5?
> >>>
> >>> Or, even better - can I just replace the 0.4 JARS in
> >>> /usr/lib/hive/share/hcatalog with the new 0.5 JARs and restart the
> >>> metastore?
> >>
> >>
> >> The upgrade procedure will differ based on how you have things deployed,
> >> primarily if you run the HiveMetaStore as a service or you directly
> connect
> >> to the HMS. You might try running a job with just the HCatalog 0.5
> client
> >> jars against your existing metastore and see if that works for you. As
> the
> >> vast majority of HCatalog is client-side, its possible to run jobs with
> >> different versions while you work out the upgrade procedure for your
> site.
> >>
> >> --travis
> >>
> >>
> >>
> >>>
> >>> Cheers,
> >>> Tim
> >>>
> >>> On Thu, Jan 24, 2013 at 5:00 PM, Travis Crawford
> >>> <traviscrawford@gmail.com> wrote:
> >>>>
> >>>> Hey Timothy -
> >>>>
> >>>> On a clean master, this command works for me:
> >>>>
> >>>>     ant clean package -Dmvn.hadoop.profile=hadoop23
> >>>>
> >>>> After doing so, we see the following jars were used in the build:
> >>>>
> >>>> fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
> >>>> core/build/lib/compile/hadoop-annotations-0.23.3.jar
> >>>> core/build/lib/compile/hadoop-archives-0.23.3.jar
> >>>> core/build/lib/compile/hadoop-auth-0.23.3.jar
> >>>> core/build/lib/compile/hadoop-common-0.23.3.jar
> >>>> core/build/lib/compile/hadoop-hdfs-0.23.3.jar
> >>>> core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
> >>>> core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
> >>>> core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
> >>>> fynn-2:hcatalog travis$
> >>>>
> >>>> I'm not familiar with how CDH4 packages things. Let's see what jars
> >>>> you're pulling in first. Also, are you changing this line in the
> top-level
> >>>> pom.xml?
> >>>>
> >>>> <hadoop23.version>0.23.3</hadoop23.version>
> >>>>
> >>>> If not, I'm surprised you see issues as you wouldn't even be compiling
> >>>> against CDH4.
> >>>>
> >>>> --travis
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter <thelabdude@gmail.com
> >
> >>>> wrote:
> >>>>>
> >>>>> Would like to upgrade to 0.5 as we're maintaining 0.4 + many of
the
> >>>>> patches supplied since the 0.4 release.
> >>>>>
> >>>>> However, I need HCatalog to run against CDH4.1.2 which based on
this
> >>>>> link
> >>>>> (
> https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647
> )
> >>>>> is hadoop-2.0.0+5

Mime
View raw message