incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sushanth Sowmyan <khorg...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 19:12:24 GMT
Sounds good to me.

On Fri, Jan 25, 2013 at 10:35 AM, Travis Crawford
<traviscrawford@gmail.com> wrote:
> Sounds good - fixes for the RC issues identified thus far have been
> reviewed. I'll get them into trunk/branch-0.5 and post a new RC.
>
> --travis
>
>
> On Fri, Jan 25, 2013 at 10:31 AM, Alan Gates <gates@hortonworks.com> wrote:
>>
>>
>> On Jan 25, 2013, at 9:18 AM, Travis Crawford wrote:
>>
>> > On Thu, Jan 24, 2013 at 7:36 PM, Sushanth Sowmyan <khorgath@gmail.com>
>> > wrote:
>> > Hmm, looks like the pom.xml used in that bigtop build is different
>> > from the one we have in our incubating candidate (probably older?) -
>> > it was bringing in hadoop-core, hadoop-test and hadoop-tools as
>> > dependencies from the root pom.xml with version ${hadoop20.version}.
>> >
>> > However, I do still see that webhcat-svr has a dependency on
>> > hadoop-core-${hadoop20.version} in the incubating candidate. We may
>> > have to wrap that inside a profile.
>> >
>> > This is intentional because webhcat does not yet build with hadoop 23. I
>> > believe there are a couple other submodules with this issue as well.
>> >
>> > Do we see this as a blocker for the 0.5 release? I don't believe full
>> > hadoop 23 support is expected yet, and I'd prefer not to block the release
>> > on this. I totally agree full 23 support is important, but think we should
>> > save it for a future release.
>>
>> +1.  We have some great new functionality.  Let's get it out there.
>>
>> Alan.
>>
>> >
>> > Thoughts?
>> >
>> > --travis
>> >
>> >
>> >
>> >
>> > On Thu, Jan 24, 2013 at 7:09 PM, Travis Crawford
>> > <traviscrawford@gmail.com> wrote:
>> > > Huh that's very interesting. If you do a verbose build maven will
>> > > print out
>> > > a dep tree and we can see why it's pulled in. Can you reproduce?
>> > >
>> > > --Travis
>> > >
>> > >
>> > > On Thursday, January 24, 2013, Sushanth Sowmyan wrote:
>> > >>
>> > >> And also, in this case, in the core/build/lib/compile/ directory, I
>> > >> see the following versions of hadoop jars (notice hadoop-core-1.0.3
>> > >> in
>> > >> the midst of the others):
>> > >>
>> > >> --
>> > >> hadoop-annotations-2.0.2.22-alpha.jar
>> > >> hadoop-archives-2.0.2.22-alpha.jar
>> > >> hadoop-auth-2.0.2.22-alpha.jar
>> > >> hadoop-common-2.0.2.22-alpha.jar
>> > >> hadoop-core-1.0.3.jar
>> > >> hadoop-hdfs-2.0.2.22-alpha.jar
>> > >> hadoop-mapreduce-client-core-2.0.2.22-alpha.jar
>> > >> hadoop-yarn-api-2.0.2.22-alpha.jar
>> > >> hadoop-yarn-common-2.0.2.22-alpha.jar
>> > >> --
>> > >>
>> > >> On Thu, Jan 24, 2013 at 5:03 PM, Sushanth Sowmyan
>> > >> <khorgath@gmail.com>
>> > >> wrote:
>> > >> > Interestingly, I'm hitting the same issue with a bigtop build
for
>> > >> > hadoop-2. And this one isn't solved by doing a clean.
>> > >> >
>> > >> > On Thu, Jan 24, 2013 at 4:26 PM, Travis Crawford
>> > >> > <traviscrawford@gmail.com> wrote:
>> > >> >> On Thu, Jan 24, 2013 at 4:20 PM, Timothy Potter
>> > >> >> <thelabdude@gmail.com>
>> > >> >> wrote:
>> > >> >>>
>> > >> >>> Hi Travis,
>> > >> >>>
>> > >> >>> Thank you! Using your ant command, I have a build.
>> > >> >>
>> > >> >>
>> > >> >> Do you know what changed? You might have had 0.20 hadoop jars
>> > >> >> already
>> > >> >> resolved, which would cause a build failure without cleaning.
>> > >> >>
>> > >> >>
>> > >> >>>
>> > >> >>> Are there any instructions or known pitfalls when upgrading
from
>> > >> >>> 0.4
>> > >> >>> to
>> > >> >>> 0.5?
>> > >> >>>
>> > >> >>> Or, even better - can I just replace the 0.4 JARS in
>> > >> >>> /usr/lib/hive/share/hcatalog with the new 0.5 JARs and
restart
>> > >> >>> the
>> > >> >>> metastore?
>> > >> >>
>> > >> >>
>> > >> >> The upgrade procedure will differ based on how you have things
>> > >> >> deployed,
>> > >> >> primarily if you run the HiveMetaStore as a service or you
>> > >> >> directly
>> > >> >> connect
>> > >> >> to the HMS. You might try running a job with just the HCatalog
0.5
>> > >> >> client
>> > >> >> jars against your existing metastore and see if that works
for
>> > >> >> you. As
>> > >> >> the
>> > >> >> vast majority of HCatalog is client-side, its possible to
run jobs
>> > >> >> with
>> > >> >> different versions while you work out the upgrade procedure
for
>> > >> >> your
>> > >> >> site.
>> > >> >>
>> > >> >> --travis
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>>
>> > >> >>> Cheers,
>> > >> >>> Tim
>> > >> >>>
>> > >> >>> On Thu, Jan 24, 2013 at 5:00 PM, Travis Crawford
>> > >> >>> <traviscrawford@gmail.com> wrote:
>> > >> >>>>
>> > >> >>>> Hey Timothy -
>> > >> >>>>
>> > >> >>>> On a clean master, this command works for me:
>> > >> >>>>
>> > >> >>>>     ant clean package -Dmvn.hadoop.profile=hadoop23
>> > >> >>>>
>> > >> >>>> After doing so, we see the following jars were used
in the
>> > >> >>>> build:
>> > >> >>>>
>> > >> >>>> fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
>> > >> >>>> core/build/lib/compile/hadoop-annotations-0.23.3.jar
>> > >> >>>> core/build/lib/compile/hadoop-archives-0.23.3.jar
>> > >> >>>> core/build/lib/compile/hadoop-auth-0.23.3.jar
>> > >> >>>> core/build/lib/compile/hadoop-common-0.23.3.jar
>> > >> >>>> core/build/lib/compile/hadoop-hdfs-0.23.3.jar
>> > >> >>>> core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
>> > >> >>>> core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
>> > >> >>>> core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
>> > >> >>>> fynn-2:hcatalog travis$
>> > >> >>>>
>> > >> >>>> I'm not familiar with how CDH4 packages things. Let's
see what
>> > >> >>>> jars
>> > >> >>>> you're pulling in first. Also, are you changing this
line in the
>> > >> >>>> top-level
>> > >> >>>> pom.xml?
>> > >> >>>>
>> > >> >>>> <hadoop23.version>0.23.3</hadoop23.version>
>> > >> >>>>
>> > >> >>>> If not, I'm surprised you see issues as you wouldn't
even be
>> > >> >>>> compiling
>> > >> >>>> against CDH4.
>> > >> >>>>
>> > >> >>>> --travis
>> > >> >>>>
>> > >> >>>>
>> > >> >>>>
>> > >> >>>>
>> > >> >>>> On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter
>> > >> >>>> <thelabdude@gmail.com>
>> > >> >>>> wrote:
>> > >> >>>>>
>> > >> >>>>> Would like to upgrade to 0.5 as we're maintaining
0.4 + many of
>> > >> >>>>> the
>> > >> >>>>> patches supplied since the 0.4 release.
>> > >> >>>>>
>> > >> >>>>> However, I need HCatalog to run against CDH4.1.2
which based on
>> > >> >>>>> this
>> > >> >>>>> link
>> > >> >>>>>
>> > >> >>>>>
>> > >> >>>>> (https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647)
>> > >> >>>>> is hadoop-2.0.0+5
>> >
>>
>

Mime
View raw message