incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Potter <thelabd...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 00:20:25 GMT
Hi Travis,

Thank you! Using your ant command, I have a build.

Are there any instructions or known pitfalls when upgrading from 0.4 to
0.5?

Or, even better - can I just replace the 0.4 JARS in
/usr/lib/hive/share/hcatalog with the new 0.5 JARs and restart the
metastore?

Cheers,
Tim

On Thu, Jan 24, 2013 at 5:00 PM, Travis Crawford
<traviscrawford@gmail.com>wrote:

> Hey Timothy -
>
> On a clean master, this command works for me:
>
>     ant clean package -Dmvn.hadoop.profile=hadoop23
>
> After doing so, we see the following jars were used in the build:
>
> fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
> core/build/lib/compile/hadoop-annotations-0.23.3.jar
> core/build/lib/compile/hadoop-archives-0.23.3.jar
> core/build/lib/compile/hadoop-auth-0.23.3.jar
> core/build/lib/compile/hadoop-common-0.23.3.jar
> core/build/lib/compile/hadoop-hdfs-0.23.3.jar
> core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
> core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
> core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
> fynn-2:hcatalog travis$
>
> I'm not familiar with how CDH4 packages things. Let's see what jars you're
> pulling in first. Also, are you changing this line in the top-level pom.xml?
>
> <hadoop23.version>0.23.3</hadoop23.version>
>
> If not, I'm surprised you see issues as you wouldn't even be compiling
> against CDH4.
>
> --travis
>
>
>
>
> On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter <thelabdude@gmail.com>wrote:
>
>> Would like to upgrade to 0.5 as we're maintaining 0.4 + many of the
>> patches supplied since the 0.4 release.
>>
>> However, I need HCatalog to run against CDH4.1.2 which based on this link
>> (
>> https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647)
>> is hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+126 ...
>>
>> What is the proper mechanism for building this to run on later versions
>> of Hadoop? The biggest issue is TaskAttemptContext and JobContext are now
>> interfaces in later versions of Hadoop but used to be concrete Java
>> classes. I think the shims code is trying to insulate us from these issues
>> but when I do:
>>
>> ant -Dmvn.hadoop.profile=hadoop23
>>
>> I get compile errors in the shims (see below).
>>
>> With 0.4 - I hacked up all the build files and some of the code to get a
>> working version. It's looking like I'm going to need to do that again?
>>
>> Here's compile issues when doing the hadoop23 build:
>>
>> jar:
>>      [echo] shims
>>     [javac] Compiling 1 source file to
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/core/build/classes
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:34:
>> package org.apache.hadoop.mapreduce.task does not exist
>>     [javac] import org.apache.hadoop.mapreduce.task.JobContextImpl;
>>     [javac]                                        ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:40:
>> cannot find symbol
>>     [javac] symbol  : class MRJobConfig
>>     [javac] location: package org.apache.hadoop.mapreduce
>>     [javac] import org.apache.hadoop.mapreduce.MRJobConfig;
>>     [javac]                                   ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:46:
>> cannot find symbol
>>     [javac] symbol  : constructor
>> TaskID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int)
>>     [javac] location: class org.apache.hadoop.mapreduce.TaskID
>>     [javac]         return new TaskID("", 0, TaskType.MAP, 0);
>>     [javac]                ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:51:
>> cannot find symbol
>>     [javac] symbol  : constructor
>> TaskAttemptID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int,int)
>>     [javac] location: class org.apache.hadoop.mapreduce.TaskAttemptID
>>     [javac]         return new TaskAttemptID("", 0, TaskType.MAP, 0, 0);
>>     [javac]                ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:57:
>> package org.apache.hadoop.mapreduce.task does not exist
>>     [javac]         return new
>> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(
>>     [javac]                                                    ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:67:
>> cannot find symbol
>>     [javac] symbol  : class TaskAttemptContextImpl
>>     [javac] location: package org.apache.hadoop.mapred
>>     [javac]             java.lang.reflect.Constructor construct =
>> org.apache.hadoop.mapred.TaskAttemptContextImpl.class.getDeclaredConstructor(
>>     [javac]
>>                 ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:82:
>> cannot find symbol
>>     [javac] symbol  : class JobContextImpl
>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>     [javac]         return new JobContextImpl(conf instanceof JobConf?
>> new JobConf(conf) : conf,
>>     [javac]                    ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:89:
>> cannot find symbol
>>     [javac] symbol  : class JobContextImpl
>>     [javac] location: package org.apache.hadoop.mapred
>>     [javac]         return new org.apache.hadoop.mapred.JobContextImpl(
>>     [javac]                                            ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:114:
>> cannot find symbol
>>     [javac] symbol  : variable MRJobConfig
>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>     [javac]             return MRJobConfig.CACHE_ARCHIVES;
>>     [javac]                    ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:116:
>> cannot find symbol
>>     [javac] symbol  : variable MRJobConfig
>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>     [javac]             return MRJobConfig.CACHE_FILES;
>>     [javac]                    ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:118:
>> cannot find symbol
>>     [javac] symbol  : variable MRJobConfig
>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>     [javac]             return MRJobConfig.CACHE_SYMLINK;
>>     [javac]                    ^
>>     [javac]
>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:128:
>> cannot find symbol
>>     [javac] symbol  : method resolvePath(org.apache.hadoop.fs.Path)
>>     [javac] location: class org.apache.hadoop.fs.FileSystem
>>     [javac]         return
>> "hdfs".equals(fs.resolvePath(path).toUri().getScheme());
>>     [javac]                                ^
>>     [javac] 12 errors
>>
>>
>> On Thu, Jan 24, 2013 at 10:49 AM, Travis Crawford <
>> traviscrawford@gmail.com> wrote:
>>
>>> On Thu, Jan 24, 2013 at 9:47 AM, Alan Gates <gates@hortonworks.com>wrote:
>>>
>>>> If it's not too much trouble the rat excludes file sounds good so that
>>>> every time we release we don't have to remember that these files are ok
>>>> without a header.
>>>>
>>>
>>> Should be easy - I'll take a look at this too.
>>>
>>> --travis
>>>
>>>
>>>
>>>>  Alan.
>>>>
>>>> On Jan 24, 2013, at 9:45 AM, Travis Crawford wrote:
>>>>
>>>> > On Thu, Jan 24, 2013 at 9:38 AM, Alan Gates <gates@hortonworks.com>
>>>> wrote:
>>>> >
>>>> > On Jan 24, 2013, at 8:56 AM, Travis Crawford wrote:
>>>> >
>>>> > > Before simply excluding these files, this is a great opportunity
to
>>>> understand what these files are and potentially fix.
>>>> > >
>>>> > > (a) The .rej file was simply an error and I will remove. I believe
>>>> it was produced when applying HCATALOG-605, which has different version
>>>> numbers in trunk/release branches.
>>>> > >
>>>> > > (b) rcfiles - should these be checked in at all? These seem like
>>>> files that should be generated when running tests, not checked into the
>>>> source tree. Can someone with more familiarity with the e2e test data clue
>>>> us into if generating makes sense?
>>>> >
>>>> > I agree these should be generated rather than checked in.  Someone
>>>> just needs to write the code to generate rcfiles during the deploy phase.
>>>>  We should file a JIRA on this.  But I don't think we should delay the
>>>> release over it.
>>>> >
>>>> > Agreed - I filed https://issues.apache.org/jira/browse/HCATALOG-607to
fix in the future but not block the release over.
>>>> >
>>>> > Do we add a rat excludes file, or let these be listed in the report?
>>>> I'll update the release docs for next time too.
>>>> >
>>>> >
>>>> > >
>>>> > > (c) text data files - we could add the header to these files, and
>>>> update whatever reads them to ignore comment lines.
>>>> >
>>>> > There's no need to put headers in these.  Data files are a well
>>>> recognized exception in license headers.
>>>> >
>>>> > Same question as above - for files that do not need a header, do we
>>>> let them be listed in the report or explicitly exclude them?
>>>> >
>>>> >
>>>> > >
>>>> > > (d) storage handler test includes/excludes - these are still around
>>>> because storagehendlers has not been migrated to the new build templates.
>>>> Unless there are objections I'd like to remove these files entirely. If we
>>>> want to ignore a test in the future it would need to have an @Ignore
>>>> annotation.
>>>> >
>>>> > +1, since excluded-tests is empty.
>>>> >
>>>> > I'm running tests for
>>>> https://issues.apache.org/jira/browse/HCATALOG-606 now and will post
>>>> the patch shortly.
>>>> >
>>>> > --travis
>>>> >
>>>> >
>>>> > >
>>>> > > Thoughts?
>>>> > >
>>>> > > --travis
>>>> > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > On Thu, Jan 24, 2013 at 1:08 AM, Sushanth Sowmyan <
>>>> khorgath@gmail.com> wrote:
>>>> > > Also, the release audit lists 7 other unapproved license :
>>>> > >
>>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/releaseaudit_report.txt
>>>> > >
>>>> > >
>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/boolean.rcfile
>>>> > >
>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/complex.rcfile
>>>> > >
>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.rcfile
>>>> > >
>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.txt
>>>> > >
>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/templeton/inpdir/nums.txt
>>>> > >
>>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/all-tests
>>>> > >
>>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/excluded-tests
>>>> > >
>>>> > > We need to add these to -e patterns for the releaseaudit target.
>>>> > >
>>>> > > -Sushanth
>>>> > >
>>>> > > On Wed, Jan 23, 2013 at 5:22 PM, Daniel Dai <daijy@hortonworks.com>
>>>> wrote:
>>>> > > > I see a release.sh.rej in the package.
>>>> > > >
>>>> > > > Thanks,
>>>> > > > Daniel
>>>> > > >
>>>> > > > On Wed, Jan 23, 2013 at 2:03 PM, Travis Crawford
>>>> > > > <traviscrawford@gmail.com> wrote:
>>>> > > >> Hi,
>>>> > > >>
>>>> > > >> I have created a candidate build for HCatalog 0.5.0-incubating.
>>>> > > >>
>>>> > > >> Keys used to sign the release are available at
>>>> > > >>
>>>> http://svn.apache.org/viewvc/incubator/hcatalog/trunk/KEYS?view=markup
>>>> > > >>
>>>> > > >> Please download, test, and try it out:
>>>> > > >>
>>>> > > >>
>>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/
>>>> > > >>
>>>> > > >> The release, md5 signature, gpg signature, and rat report
can
>>>> all be found
>>>> > > >> at the above address.
>>>> > > >>
>>>> > > >> Should we release this? Vote closes on Monday, January
28.
>>>> > > >>
>>>> > > >> --travis
>>>> > >
>>>> >
>>>> >
>>>>
>>>>
>>>
>>
>

Mime
View raw message