incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Crawford <traviscrawf...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 00:00:57 GMT
Hey Timothy -

On a clean master, this command works for me:

    ant clean package -Dmvn.hadoop.profile=hadoop23

After doing so, we see the following jars were used in the build:

fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
core/build/lib/compile/hadoop-annotations-0.23.3.jar
core/build/lib/compile/hadoop-archives-0.23.3.jar
core/build/lib/compile/hadoop-auth-0.23.3.jar
core/build/lib/compile/hadoop-common-0.23.3.jar
core/build/lib/compile/hadoop-hdfs-0.23.3.jar
core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
fynn-2:hcatalog travis$

I'm not familiar with how CDH4 packages things. Let's see what jars you're
pulling in first. Also, are you changing this line in the top-level pom.xml?

<hadoop23.version>0.23.3</hadoop23.version>

If not, I'm surprised you see issues as you wouldn't even be compiling
against CDH4.

--travis




On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter <thelabdude@gmail.com>wrote:

> Would like to upgrade to 0.5 as we're maintaining 0.4 + many of the
> patches supplied since the 0.4 release.
>
> However, I need HCatalog to run against CDH4.1.2 which based on this link (
> https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647)
> is hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+126 ...
>
> What is the proper mechanism for building this to run on later versions of
> Hadoop? The biggest issue is TaskAttemptContext and JobContext are now
> interfaces in later versions of Hadoop but used to be concrete Java
> classes. I think the shims code is trying to insulate us from these issues
> but when I do:
>
> ant -Dmvn.hadoop.profile=hadoop23
>
> I get compile errors in the shims (see below).
>
> With 0.4 - I hacked up all the build files and some of the code to get a
> working version. It's looking like I'm going to need to do that again?
>
> Here's compile issues when doing the hadoop23 build:
>
> jar:
>      [echo] shims
>     [javac] Compiling 1 source file to
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/core/build/classes
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:34:
> package org.apache.hadoop.mapreduce.task does not exist
>     [javac] import org.apache.hadoop.mapreduce.task.JobContextImpl;
>     [javac]                                        ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:40:
> cannot find symbol
>     [javac] symbol  : class MRJobConfig
>     [javac] location: package org.apache.hadoop.mapreduce
>     [javac] import org.apache.hadoop.mapreduce.MRJobConfig;
>     [javac]                                   ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:46:
> cannot find symbol
>     [javac] symbol  : constructor
> TaskID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int)
>     [javac] location: class org.apache.hadoop.mapreduce.TaskID
>     [javac]         return new TaskID("", 0, TaskType.MAP, 0);
>     [javac]                ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:51:
> cannot find symbol
>     [javac] symbol  : constructor
> TaskAttemptID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int,int)
>     [javac] location: class org.apache.hadoop.mapreduce.TaskAttemptID
>     [javac]         return new TaskAttemptID("", 0, TaskType.MAP, 0, 0);
>     [javac]                ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:57:
> package org.apache.hadoop.mapreduce.task does not exist
>     [javac]         return new
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(
>     [javac]                                                    ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:67:
> cannot find symbol
>     [javac] symbol  : class TaskAttemptContextImpl
>     [javac] location: package org.apache.hadoop.mapred
>     [javac]             java.lang.reflect.Constructor construct =
> org.apache.hadoop.mapred.TaskAttemptContextImpl.class.getDeclaredConstructor(
>     [javac]
>                 ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:82:
> cannot find symbol
>     [javac] symbol  : class JobContextImpl
>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>     [javac]         return new JobContextImpl(conf instanceof JobConf? new
> JobConf(conf) : conf,
>     [javac]                    ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:89:
> cannot find symbol
>     [javac] symbol  : class JobContextImpl
>     [javac] location: package org.apache.hadoop.mapred
>     [javac]         return new org.apache.hadoop.mapred.JobContextImpl(
>     [javac]                                            ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:114:
> cannot find symbol
>     [javac] symbol  : variable MRJobConfig
>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>     [javac]             return MRJobConfig.CACHE_ARCHIVES;
>     [javac]                    ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:116:
> cannot find symbol
>     [javac] symbol  : variable MRJobConfig
>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>     [javac]             return MRJobConfig.CACHE_FILES;
>     [javac]                    ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:118:
> cannot find symbol
>     [javac] symbol  : variable MRJobConfig
>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>     [javac]             return MRJobConfig.CACHE_SYMLINK;
>     [javac]                    ^
>     [javac]
> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:128:
> cannot find symbol
>     [javac] symbol  : method resolvePath(org.apache.hadoop.fs.Path)
>     [javac] location: class org.apache.hadoop.fs.FileSystem
>     [javac]         return
> "hdfs".equals(fs.resolvePath(path).toUri().getScheme());
>     [javac]                                ^
>     [javac] 12 errors
>
>
> On Thu, Jan 24, 2013 at 10:49 AM, Travis Crawford <
> traviscrawford@gmail.com> wrote:
>
>> On Thu, Jan 24, 2013 at 9:47 AM, Alan Gates <gates@hortonworks.com>wrote:
>>
>>> If it's not too much trouble the rat excludes file sounds good so that
>>> every time we release we don't have to remember that these files are ok
>>> without a header.
>>>
>>
>> Should be easy - I'll take a look at this too.
>>
>> --travis
>>
>>
>>
>>>  Alan.
>>>
>>> On Jan 24, 2013, at 9:45 AM, Travis Crawford wrote:
>>>
>>> > On Thu, Jan 24, 2013 at 9:38 AM, Alan Gates <gates@hortonworks.com>
>>> wrote:
>>> >
>>> > On Jan 24, 2013, at 8:56 AM, Travis Crawford wrote:
>>> >
>>> > > Before simply excluding these files, this is a great opportunity to
>>> understand what these files are and potentially fix.
>>> > >
>>> > > (a) The .rej file was simply an error and I will remove. I believe
>>> it was produced when applying HCATALOG-605, which has different version
>>> numbers in trunk/release branches.
>>> > >
>>> > > (b) rcfiles - should these be checked in at all? These seem like
>>> files that should be generated when running tests, not checked into the
>>> source tree. Can someone with more familiarity with the e2e test data clue
>>> us into if generating makes sense?
>>> >
>>> > I agree these should be generated rather than checked in.  Someone
>>> just needs to write the code to generate rcfiles during the deploy phase.
>>>  We should file a JIRA on this.  But I don't think we should delay the
>>> release over it.
>>> >
>>> > Agreed - I filed https://issues.apache.org/jira/browse/HCATALOG-607to fix
in the future but not block the release over.
>>> >
>>> > Do we add a rat excludes file, or let these be listed in the report?
>>> I'll update the release docs for next time too.
>>> >
>>> >
>>> > >
>>> > > (c) text data files - we could add the header to these files, and
>>> update whatever reads them to ignore comment lines.
>>> >
>>> > There's no need to put headers in these.  Data files are a well
>>> recognized exception in license headers.
>>> >
>>> > Same question as above - for files that do not need a header, do we
>>> let them be listed in the report or explicitly exclude them?
>>> >
>>> >
>>> > >
>>> > > (d) storage handler test includes/excludes - these are still around
>>> because storagehendlers has not been migrated to the new build templates.
>>> Unless there are objections I'd like to remove these files entirely. If we
>>> want to ignore a test in the future it would need to have an @Ignore
>>> annotation.
>>> >
>>> > +1, since excluded-tests is empty.
>>> >
>>> > I'm running tests for
>>> https://issues.apache.org/jira/browse/HCATALOG-606 now and will post
>>> the patch shortly.
>>> >
>>> > --travis
>>> >
>>> >
>>> > >
>>> > > Thoughts?
>>> > >
>>> > > --travis
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Thu, Jan 24, 2013 at 1:08 AM, Sushanth Sowmyan <
>>> khorgath@gmail.com> wrote:
>>> > > Also, the release audit lists 7 other unapproved license :
>>> > >
>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/releaseaudit_report.txt
>>> > >
>>> > >
>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/boolean.rcfile
>>> > >
>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/complex.rcfile
>>> > >
>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.rcfile
>>> > >
>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.txt
>>> > >
>>> hcatalog-src-0.5.0-incubating/src/test/e2e/templeton/inpdir/nums.txt
>>> > >
>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/all-tests
>>> > >
>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/excluded-tests
>>> > >
>>> > > We need to add these to -e patterns for the releaseaudit target.
>>> > >
>>> > > -Sushanth
>>> > >
>>> > > On Wed, Jan 23, 2013 at 5:22 PM, Daniel Dai <daijy@hortonworks.com>
>>> wrote:
>>> > > > I see a release.sh.rej in the package.
>>> > > >
>>> > > > Thanks,
>>> > > > Daniel
>>> > > >
>>> > > > On Wed, Jan 23, 2013 at 2:03 PM, Travis Crawford
>>> > > > <traviscrawford@gmail.com> wrote:
>>> > > >> Hi,
>>> > > >>
>>> > > >> I have created a candidate build for HCatalog 0.5.0-incubating.
>>> > > >>
>>> > > >> Keys used to sign the release are available at
>>> > > >>
>>> http://svn.apache.org/viewvc/incubator/hcatalog/trunk/KEYS?view=markup
>>> > > >>
>>> > > >> Please download, test, and try it out:
>>> > > >>
>>> > > >>
>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/
>>> > > >>
>>> > > >> The release, md5 signature, gpg signature, and rat report
can all
>>> be found
>>> > > >> at the above address.
>>> > > >>
>>> > > >> Should we release this? Vote closes on Monday, January 28.
>>> > > >>
>>> > > >> --travis
>>> > >
>>> >
>>> >
>>>
>>>
>>
>

Mime
View raw message