Hey Timothy -

On a clean master, this command works for me:

    ant clean package -Dmvn.hadoop.profile=hadoop23

After doing so, we see the following jars were used in the build:

fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
core/build/lib/compile/hadoop-annotations-0.23.3.jar
core/build/lib/compile/hadoop-archives-0.23.3.jar
core/build/lib/compile/hadoop-auth-0.23.3.jar
core/build/lib/compile/hadoop-common-0.23.3.jar
core/build/lib/compile/hadoop-hdfs-0.23.3.jar
core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
fynn-2:hcatalog travis$ 

I'm not familiar with how CDH4 packages things. Let's see what jars you're pulling in first. Also, are you changing this line in the top-level pom.xml?

<hadoop23.version>0.23.3</hadoop23.version>

If not, I'm surprised you see issues as you wouldn't even be compiling against CDH4.

--travis




On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter <thelabdude@gmail.com> wrote:
Would like to upgrade to 0.5 as we're maintaining 0.4 + many of the patches supplied since the 0.4 release. 

However, I need HCatalog to run against CDH4.1.2 which based on this link (https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647) is hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+126 ... 

What is the proper mechanism for building this to run on later versions of Hadoop? The biggest issue is TaskAttemptContext and JobContext are now interfaces in later versions of Hadoop but used to be concrete Java classes. I think the shims code is trying to insulate us from these issues but when I do:

ant -Dmvn.hadoop.profile=hadoop23

I get compile errors in the shims (see below).

With 0.4 - I hacked up all the build files and some of the code to get a working version. It's looking like I'm going to need to do that again?

Here's compile issues when doing the hadoop23 build:

jar:
     [echo] shims
    [javac] Compiling 1 source file to /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/core/build/classes
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:34: package org.apache.hadoop.mapreduce.task does not exist
    [javac] import org.apache.hadoop.mapreduce.task.JobContextImpl;
    [javac]                                        ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:40: cannot find symbol
    [javac] symbol  : class MRJobConfig
    [javac] location: package org.apache.hadoop.mapreduce
    [javac] import org.apache.hadoop.mapreduce.MRJobConfig;
    [javac]                                   ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:46: cannot find symbol
    [javac] symbol  : constructor TaskID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int)
    [javac] location: class org.apache.hadoop.mapreduce.TaskID
    [javac]         return new TaskID("", 0, TaskType.MAP, 0);
    [javac]                ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:51: cannot find symbol
    [javac] symbol  : constructor TaskAttemptID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int,int)
    [javac] location: class org.apache.hadoop.mapreduce.TaskAttemptID
    [javac]         return new TaskAttemptID("", 0, TaskType.MAP, 0, 0);
    [javac]                ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:57: package org.apache.hadoop.mapreduce.task does not exist
    [javac]         return new org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(
    [javac]                                                    ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:67: cannot find symbol
    [javac] symbol  : class TaskAttemptContextImpl
    [javac] location: package org.apache.hadoop.mapred
    [javac]             java.lang.reflect.Constructor construct = org.apache.hadoop.mapred.TaskAttemptContextImpl.class.getDeclaredConstructor(
    [javac]                                                                               ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:82: cannot find symbol
    [javac] symbol  : class JobContextImpl
    [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
    [javac]         return new JobContextImpl(conf instanceof JobConf? new JobConf(conf) : conf,
    [javac]                    ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:89: cannot find symbol
    [javac] symbol  : class JobContextImpl
    [javac] location: package org.apache.hadoop.mapred
    [javac]         return new org.apache.hadoop.mapred.JobContextImpl(
    [javac]                                            ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:114: cannot find symbol
    [javac] symbol  : variable MRJobConfig
    [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
    [javac]             return MRJobConfig.CACHE_ARCHIVES;
    [javac]                    ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:116: cannot find symbol
    [javac] symbol  : variable MRJobConfig
    [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
    [javac]             return MRJobConfig.CACHE_FILES;
    [javac]                    ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:118: cannot find symbol
    [javac] symbol  : variable MRJobConfig
    [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
    [javac]             return MRJobConfig.CACHE_SYMLINK;
    [javac]                    ^
    [javac] /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:128: cannot find symbol
    [javac] symbol  : method resolvePath(org.apache.hadoop.fs.Path)
    [javac] location: class org.apache.hadoop.fs.FileSystem
    [javac]         return "hdfs".equals(fs.resolvePath(path).toUri().getScheme());
    [javac]                                ^
    [javac] 12 errors


On Thu, Jan 24, 2013 at 10:49 AM, Travis Crawford <traviscrawford@gmail.com> wrote:
On Thu, Jan 24, 2013 at 9:47 AM, Alan Gates <gates@hortonworks.com> wrote:
If it's not too much trouble the rat excludes file sounds good so that every time we release we don't have to remember that these files are ok without a header.

Should be easy - I'll take a look at this too.

--travis

 
Alan.

On Jan 24, 2013, at 9:45 AM, Travis Crawford wrote:

> On Thu, Jan 24, 2013 at 9:38 AM, Alan Gates <gates@hortonworks.com> wrote:
>
> On Jan 24, 2013, at 8:56 AM, Travis Crawford wrote:
>
> > Before simply excluding these files, this is a great opportunity to understand what these files are and potentially fix.
> >
> > (a) The .rej file was simply an error and I will remove. I believe it was produced when applying HCATALOG-605, which has different version numbers in trunk/release branches.
> >
> > (b) rcfiles - should these be checked in at all? These seem like files that should be generated when running tests, not checked into the source tree. Can someone with more familiarity with the e2e test data clue us into if generating makes sense?
>
> I agree these should be generated rather than checked in.  Someone just needs to write the code to generate rcfiles during the deploy phase.  We should file a JIRA on this.  But I don't think we should delay the release over it.
>
> Agreed - I filed https://issues.apache.org/jira/browse/HCATALOG-607 to fix in the future but not block the release over.
>
> Do we add a rat excludes file, or let these be listed in the report? I'll update the release docs for next time too.
>
>
> >
> > (c) text data files - we could add the header to these files, and update whatever reads them to ignore comment lines.
>
> There's no need to put headers in these.  Data files are a well recognized exception in license headers.
>
> Same question as above - for files that do not need a header, do we let them be listed in the report or explicitly exclude them?
>
>
> >
> > (d) storage handler test includes/excludes - these are still around because storagehendlers has not been migrated to the new build templates. Unless there are objections I'd like to remove these files entirely. If we want to ignore a test in the future it would need to have an @Ignore annotation.
>
> +1, since excluded-tests is empty.
>
> I'm running tests for https://issues.apache.org/jira/browse/HCATALOG-606 now and will post the patch shortly.
>
> --travis
>
>
> >
> > Thoughts?
> >
> > --travis
> >
> >
> >
> >
> > On Thu, Jan 24, 2013 at 1:08 AM, Sushanth Sowmyan <khorgath@gmail.com> wrote:
> > Also, the release audit lists 7 other unapproved license :
> > http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/releaseaudit_report.txt
> >
> >   hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/boolean.rcfile
> >   hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/complex.rcfile
> >   hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.rcfile
> >   hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.txt
> >   hcatalog-src-0.5.0-incubating/src/test/e2e/templeton/inpdir/nums.txt
> >   hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/all-tests
> >   hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/excluded-tests
> >
> > We need to add these to -e patterns for the releaseaudit target.
> >
> > -Sushanth
> >
> > On Wed, Jan 23, 2013 at 5:22 PM, Daniel Dai <daijy@hortonworks.com> wrote:
> > > I see a release.sh.rej in the package.
> > >
> > > Thanks,
> > > Daniel
> > >
> > > On Wed, Jan 23, 2013 at 2:03 PM, Travis Crawford
> > > <traviscrawford@gmail.com> wrote:
> > >> Hi,
> > >>
> > >> I have created a candidate build for HCatalog 0.5.0-incubating.
> > >>
> > >> Keys used to sign the release are available at
> > >> http://svn.apache.org/viewvc/incubator/hcatalog/trunk/KEYS?view=markup
> > >>
> > >> Please download, test, and try it out:
> > >>
> > >>     http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/
> > >>
> > >> The release, md5 signature, gpg signature, and rat report can all be found
> > >> at the above address.
> > >>
> > >> Should we release this? Vote closes on Monday, January 28.
> > >>
> > >> --travis
> >
>
>