incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Potter <thelabd...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 00:39:28 GMT
Will try your suggestion, thanks! Btw ... we've been running HCatalog in a
small-medium sized production Hadoop cluster (~400 cores) for several
months. It rocks! So I know you guys say it's not ready for production yet,
but it's been awesome for us.

Cheers,
Tim

On Thu, Jan 24, 2013 at 5:26 PM, Travis Crawford
<traviscrawford@gmail.com>wrote:

> On Thu, Jan 24, 2013 at 4:20 PM, Timothy Potter <thelabdude@gmail.com>wrote:
>
>> Hi Travis,
>>
>> Thank you! Using your ant command, I have a build.
>>
>
> Do you know what changed? You might have had 0.20 hadoop jars already
> resolved, which would cause a build failure without cleaning.
>
>
>
>> Are there any instructions or known pitfalls when upgrading from 0.4 to
>> 0.5?
>>
>> Or, even better - can I just replace the 0.4 JARS in
>> /usr/lib/hive/share/hcatalog with the new 0.5 JARs and restart the
>> metastore?
>>
>
> The upgrade procedure will differ based on how you have things deployed,
> primarily if you run the HiveMetaStore as a service or you directly connect
> to the HMS. You might try running a job with just the HCatalog 0.5 client
> jars against your existing metastore and see if that works for you. As the
> vast majority of HCatalog is client-side, its possible to run jobs with
> different versions while you work out the upgrade procedure for your site.
>
> --travis
>
>
>
>
>> Cheers,
>> Tim
>>
>> On Thu, Jan 24, 2013 at 5:00 PM, Travis Crawford <
>> traviscrawford@gmail.com> wrote:
>>
>>> Hey Timothy -
>>>
>>> On a clean master, this command works for me:
>>>
>>>     ant clean package -Dmvn.hadoop.profile=hadoop23
>>>
>>> After doing so, we see the following jars were used in the build:
>>>
>>> fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
>>> core/build/lib/compile/hadoop-annotations-0.23.3.jar
>>> core/build/lib/compile/hadoop-archives-0.23.3.jar
>>> core/build/lib/compile/hadoop-auth-0.23.3.jar
>>> core/build/lib/compile/hadoop-common-0.23.3.jar
>>> core/build/lib/compile/hadoop-hdfs-0.23.3.jar
>>> core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
>>> core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
>>> core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
>>> fynn-2:hcatalog travis$
>>>
>>> I'm not familiar with how CDH4 packages things. Let's see what jars
>>> you're pulling in first. Also, are you changing this line in the top-level
>>> pom.xml?
>>>
>>> <hadoop23.version>0.23.3</hadoop23.version>
>>>
>>> If not, I'm surprised you see issues as you wouldn't even be compiling
>>> against CDH4.
>>>
>>> --travis
>>>
>>>
>>>
>>>
>>> On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter <thelabdude@gmail.com>wrote:
>>>
>>>> Would like to upgrade to 0.5 as we're maintaining 0.4 + many of the
>>>> patches supplied since the 0.4 release.
>>>>
>>>> However, I need HCatalog to run against CDH4.1.2 which based on this
>>>> link (
>>>> https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647)
>>>> is hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+126 ...
>>>>
>>>> What is the proper mechanism for building this to run on later versions
>>>> of Hadoop? The biggest issue is TaskAttemptContext and JobContext are now
>>>> interfaces in later versions of Hadoop but used to be concrete Java
>>>> classes. I think the shims code is trying to insulate us from these issues
>>>> but when I do:
>>>>
>>>> ant -Dmvn.hadoop.profile=hadoop23
>>>>
>>>> I get compile errors in the shims (see below).
>>>>
>>>> With 0.4 - I hacked up all the build files and some of the code to get
>>>> a working version. It's looking like I'm going to need to do that again?
>>>>
>>>> Here's compile issues when doing the hadoop23 build:
>>>>
>>>> jar:
>>>>      [echo] shims
>>>>     [javac] Compiling 1 source file to
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/core/build/classes
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:34:
>>>> package org.apache.hadoop.mapreduce.task does not exist
>>>>     [javac] import org.apache.hadoop.mapreduce.task.JobContextImpl;
>>>>     [javac]                                        ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:40:
>>>> cannot find symbol
>>>>     [javac] symbol  : class MRJobConfig
>>>>     [javac] location: package org.apache.hadoop.mapreduce
>>>>     [javac] import org.apache.hadoop.mapreduce.MRJobConfig;
>>>>     [javac]                                   ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:46:
>>>> cannot find symbol
>>>>     [javac] symbol  : constructor
>>>> TaskID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int)
>>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskID
>>>>     [javac]         return new TaskID("", 0, TaskType.MAP, 0);
>>>>     [javac]                ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:51:
>>>> cannot find symbol
>>>>     [javac] symbol  : constructor
>>>> TaskAttemptID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int,int)
>>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskAttemptID
>>>>     [javac]         return new TaskAttemptID("", 0, TaskType.MAP, 0, 0);
>>>>     [javac]                ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:57:
>>>> package org.apache.hadoop.mapreduce.task does not exist
>>>>     [javac]         return new
>>>> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(
>>>>     [javac]                                                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:67:
>>>> cannot find symbol
>>>>     [javac] symbol  : class TaskAttemptContextImpl
>>>>     [javac] location: package org.apache.hadoop.mapred
>>>>     [javac]             java.lang.reflect.Constructor construct =
>>>> org.apache.hadoop.mapred.TaskAttemptContextImpl.class.getDeclaredConstructor(
>>>>     [javac]
>>>>                   ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:82:
>>>> cannot find symbol
>>>>     [javac] symbol  : class JobContextImpl
>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>     [javac]         return new JobContextImpl(conf instanceof JobConf?
>>>> new JobConf(conf) : conf,
>>>>     [javac]                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:89:
>>>> cannot find symbol
>>>>     [javac] symbol  : class JobContextImpl
>>>>     [javac] location: package org.apache.hadoop.mapred
>>>>     [javac]         return new org.apache.hadoop.mapred.JobContextImpl(
>>>>     [javac]                                            ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:114:
>>>> cannot find symbol
>>>>     [javac] symbol  : variable MRJobConfig
>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>     [javac]             return MRJobConfig.CACHE_ARCHIVES;
>>>>     [javac]                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:116:
>>>> cannot find symbol
>>>>     [javac] symbol  : variable MRJobConfig
>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>     [javac]             return MRJobConfig.CACHE_FILES;
>>>>     [javac]                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:118:
>>>> cannot find symbol
>>>>     [javac] symbol  : variable MRJobConfig
>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>     [javac]             return MRJobConfig.CACHE_SYMLINK;
>>>>     [javac]                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:128:
>>>> cannot find symbol
>>>>     [javac] symbol  : method resolvePath(org.apache.hadoop.fs.Path)
>>>>     [javac] location: class org.apache.hadoop.fs.FileSystem
>>>>     [javac]         return
>>>> "hdfs".equals(fs.resolvePath(path).toUri().getScheme());
>>>>     [javac]                                ^
>>>>     [javac] 12 errors
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 10:49 AM, Travis Crawford <
>>>> traviscrawford@gmail.com> wrote:
>>>>
>>>>> On Thu, Jan 24, 2013 at 9:47 AM, Alan Gates <gates@hortonworks.com>wrote:
>>>>>
>>>>>> If it's not too much trouble the rat excludes file sounds good so
>>>>>> that every time we release we don't have to remember that these files
are
>>>>>> ok without a header.
>>>>>>
>>>>>
>>>>> Should be easy - I'll take a look at this too.
>>>>>
>>>>> --travis
>>>>>
>>>>>
>>>>>
>>>>>>  Alan.
>>>>>>
>>>>>> On Jan 24, 2013, at 9:45 AM, Travis Crawford wrote:
>>>>>>
>>>>>> > On Thu, Jan 24, 2013 at 9:38 AM, Alan Gates <gates@hortonworks.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > On Jan 24, 2013, at 8:56 AM, Travis Crawford wrote:
>>>>>> >
>>>>>> > > Before simply excluding these files, this is a great opportunity
>>>>>> to understand what these files are and potentially fix.
>>>>>> > >
>>>>>> > > (a) The .rej file was simply an error and I will remove.
I
>>>>>> believe it was produced when applying HCATALOG-605, which has different
>>>>>> version numbers in trunk/release branches.
>>>>>> > >
>>>>>> > > (b) rcfiles - should these be checked in at all? These
seem like
>>>>>> files that should be generated when running tests, not checked into
the
>>>>>> source tree. Can someone with more familiarity with the e2e test
data clue
>>>>>> us into if generating makes sense?
>>>>>> >
>>>>>> > I agree these should be generated rather than checked in.  Someone
>>>>>> just needs to write the code to generate rcfiles during the deploy
phase.
>>>>>>  We should file a JIRA on this.  But I don't think we should delay
the
>>>>>> release over it.
>>>>>> >
>>>>>> > Agreed - I filed https://issues.apache.org/jira/browse/HCATALOG-607to
fix in the future but not block the release over.
>>>>>> >
>>>>>> > Do we add a rat excludes file, or let these be listed in the
>>>>>> report? I'll update the release docs for next time too.
>>>>>> >
>>>>>> >
>>>>>> > >
>>>>>> > > (c) text data files - we could add the header to these
files, and
>>>>>> update whatever reads them to ignore comment lines.
>>>>>> >
>>>>>> > There's no need to put headers in these.  Data files are a well
>>>>>> recognized exception in license headers.
>>>>>> >
>>>>>> > Same question as above - for files that do not need a header,
do we
>>>>>> let them be listed in the report or explicitly exclude them?
>>>>>> >
>>>>>> >
>>>>>> > >
>>>>>> > > (d) storage handler test includes/excludes - these are
still
>>>>>> around because storagehendlers has not been migrated to the new build
>>>>>> templates. Unless there are objections I'd like to remove these files
>>>>>> entirely. If we want to ignore a test in the future it would need
to have
>>>>>> an @Ignore annotation.
>>>>>> >
>>>>>> > +1, since excluded-tests is empty.
>>>>>> >
>>>>>> > I'm running tests for
>>>>>> https://issues.apache.org/jira/browse/HCATALOG-606 now and will post
>>>>>> the patch shortly.
>>>>>> >
>>>>>> > --travis
>>>>>> >
>>>>>> >
>>>>>> > >
>>>>>> > > Thoughts?
>>>>>> > >
>>>>>> > > --travis
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > > On Thu, Jan 24, 2013 at 1:08 AM, Sushanth Sowmyan <
>>>>>> khorgath@gmail.com> wrote:
>>>>>> > > Also, the release audit lists 7 other unapproved license
:
>>>>>> > >
>>>>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/releaseaudit_report.txt
>>>>>> > >
>>>>>> > >
>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/boolean.rcfile
>>>>>> > >
>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/complex.rcfile
>>>>>> > >
>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.rcfile
>>>>>> > >
>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.txt
>>>>>> > >
>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/templeton/inpdir/nums.txt
>>>>>> > >
>>>>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/all-tests
>>>>>> > >
>>>>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/excluded-tests
>>>>>> > >
>>>>>> > > We need to add these to -e patterns for the releaseaudit
target.
>>>>>> > >
>>>>>> > > -Sushanth
>>>>>> > >
>>>>>> > > On Wed, Jan 23, 2013 at 5:22 PM, Daniel Dai <
>>>>>> daijy@hortonworks.com> wrote:
>>>>>> > > > I see a release.sh.rej in the package.
>>>>>> > > >
>>>>>> > > > Thanks,
>>>>>> > > > Daniel
>>>>>> > > >
>>>>>> > > > On Wed, Jan 23, 2013 at 2:03 PM, Travis Crawford
>>>>>> > > > <traviscrawford@gmail.com> wrote:
>>>>>> > > >> Hi,
>>>>>> > > >>
>>>>>> > > >> I have created a candidate build for HCatalog
0.5.0-incubating.
>>>>>> > > >>
>>>>>> > > >> Keys used to sign the release are available at
>>>>>> > > >>
>>>>>> http://svn.apache.org/viewvc/incubator/hcatalog/trunk/KEYS?view=markup
>>>>>> > > >>
>>>>>> > > >> Please download, test, and try it out:
>>>>>> > > >>
>>>>>> > > >>
>>>>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/
>>>>>> > > >>
>>>>>> > > >> The release, md5 signature, gpg signature, and
rat report can
>>>>>> all be found
>>>>>> > > >> at the above address.
>>>>>> > > >>
>>>>>> > > >> Should we release this? Vote closes on Monday,
January 28.
>>>>>> > > >>
>>>>>> > > >> --travis
>>>>>> > >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message