incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sushanth Sowmyan <khorg...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 01:03:20 GMT
Interestingly, I'm hitting the same issue with a bigtop build for
hadoop-2. And this one isn't solved by doing a clean.

On Thu, Jan 24, 2013 at 4:26 PM, Travis Crawford
<traviscrawford@gmail.com> wrote:
> On Thu, Jan 24, 2013 at 4:20 PM, Timothy Potter <thelabdude@gmail.com>
> wrote:
>>
>> Hi Travis,
>>
>> Thank you! Using your ant command, I have a build.
>
>
> Do you know what changed? You might have had 0.20 hadoop jars already
> resolved, which would cause a build failure without cleaning.
>
>
>>
>> Are there any instructions or known pitfalls when upgrading from 0.4 to
>> 0.5?
>>
>> Or, even better - can I just replace the 0.4 JARS in
>> /usr/lib/hive/share/hcatalog with the new 0.5 JARs and restart the
>> metastore?
>
>
> The upgrade procedure will differ based on how you have things deployed,
> primarily if you run the HiveMetaStore as a service or you directly connect
> to the HMS. You might try running a job with just the HCatalog 0.5 client
> jars against your existing metastore and see if that works for you. As the
> vast majority of HCatalog is client-side, its possible to run jobs with
> different versions while you work out the upgrade procedure for your site.
>
> --travis
>
>
>
>>
>> Cheers,
>> Tim
>>
>> On Thu, Jan 24, 2013 at 5:00 PM, Travis Crawford
>> <traviscrawford@gmail.com> wrote:
>>>
>>> Hey Timothy -
>>>
>>> On a clean master, this command works for me:
>>>
>>>     ant clean package -Dmvn.hadoop.profile=hadoop23
>>>
>>> After doing so, we see the following jars were used in the build:
>>>
>>> fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
>>> core/build/lib/compile/hadoop-annotations-0.23.3.jar
>>> core/build/lib/compile/hadoop-archives-0.23.3.jar
>>> core/build/lib/compile/hadoop-auth-0.23.3.jar
>>> core/build/lib/compile/hadoop-common-0.23.3.jar
>>> core/build/lib/compile/hadoop-hdfs-0.23.3.jar
>>> core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
>>> core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
>>> core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
>>> fynn-2:hcatalog travis$
>>>
>>> I'm not familiar with how CDH4 packages things. Let's see what jars
>>> you're pulling in first. Also, are you changing this line in the top-level
>>> pom.xml?
>>>
>>> <hadoop23.version>0.23.3</hadoop23.version>
>>>
>>> If not, I'm surprised you see issues as you wouldn't even be compiling
>>> against CDH4.
>>>
>>> --travis
>>>
>>>
>>>
>>>
>>> On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter <thelabdude@gmail.com>
>>> wrote:
>>>>
>>>> Would like to upgrade to 0.5 as we're maintaining 0.4 + many of the
>>>> patches supplied since the 0.4 release.
>>>>
>>>> However, I need HCatalog to run against CDH4.1.2 which based on this
>>>> link
>>>> (https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647)
>>>> is hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+126 ...
>>>>
>>>> What is the proper mechanism for building this to run on later versions
>>>> of Hadoop? The biggest issue is TaskAttemptContext and JobContext are now
>>>> interfaces in later versions of Hadoop but used to be concrete Java classes.
>>>> I think the shims code is trying to insulate us from these issues but when
I
>>>> do:
>>>>
>>>> ant -Dmvn.hadoop.profile=hadoop23
>>>>
>>>> I get compile errors in the shims (see below).
>>>>
>>>> With 0.4 - I hacked up all the build files and some of the code to get a
>>>> working version. It's looking like I'm going to need to do that again?
>>>>
>>>> Here's compile issues when doing the hadoop23 build:
>>>>
>>>> jar:
>>>>      [echo] shims
>>>>     [javac] Compiling 1 source file to
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/core/build/classes
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:34:
>>>> package org.apache.hadoop.mapreduce.task does not exist
>>>>     [javac] import org.apache.hadoop.mapreduce.task.JobContextImpl;
>>>>     [javac]                                        ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:40:
>>>> cannot find symbol
>>>>     [javac] symbol  : class MRJobConfig
>>>>     [javac] location: package org.apache.hadoop.mapreduce
>>>>     [javac] import org.apache.hadoop.mapreduce.MRJobConfig;
>>>>     [javac]                                   ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:46:
>>>> cannot find symbol
>>>>     [javac] symbol  : constructor
>>>> TaskID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int)
>>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskID
>>>>     [javac]         return new TaskID("", 0, TaskType.MAP, 0);
>>>>     [javac]                ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:51:
>>>> cannot find symbol
>>>>     [javac] symbol  : constructor
>>>> TaskAttemptID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int,int)
>>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskAttemptID
>>>>     [javac]         return new TaskAttemptID("", 0, TaskType.MAP, 0, 0);
>>>>     [javac]                ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:57:
>>>> package org.apache.hadoop.mapreduce.task does not exist
>>>>     [javac]         return new
>>>> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(
>>>>     [javac]                                                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:67:
>>>> cannot find symbol
>>>>     [javac] symbol  : class TaskAttemptContextImpl
>>>>     [javac] location: package org.apache.hadoop.mapred
>>>>     [javac]             java.lang.reflect.Constructor construct =
>>>> org.apache.hadoop.mapred.TaskAttemptContextImpl.class.getDeclaredConstructor(
>>>>     [javac]
>>>> ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:82:
>>>> cannot find symbol
>>>>     [javac] symbol  : class JobContextImpl
>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>     [javac]         return new JobContextImpl(conf instanceof JobConf?
>>>> new JobConf(conf) : conf,
>>>>     [javac]                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:89:
>>>> cannot find symbol
>>>>     [javac] symbol  : class JobContextImpl
>>>>     [javac] location: package org.apache.hadoop.mapred
>>>>     [javac]         return new org.apache.hadoop.mapred.JobContextImpl(
>>>>     [javac]                                            ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:114:
>>>> cannot find symbol
>>>>     [javac] symbol  : variable MRJobConfig
>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>     [javac]             return MRJobConfig.CACHE_ARCHIVES;
>>>>     [javac]                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:116:
>>>> cannot find symbol
>>>>     [javac] symbol  : variable MRJobConfig
>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>     [javac]             return MRJobConfig.CACHE_FILES;
>>>>     [javac]                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:118:
>>>> cannot find symbol
>>>>     [javac] symbol  : variable MRJobConfig
>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>     [javac]             return MRJobConfig.CACHE_SYMLINK;
>>>>     [javac]                    ^
>>>>     [javac]
>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:128:
>>>> cannot find symbol
>>>>     [javac] symbol  : method resolvePath(org.apache.hadoop.fs.Path)
>>>>     [javac] location: class org.apache.hadoop.fs.FileSystem
>>>>     [javac]         return
>>>> "hdfs".equals(fs.resolvePath(path).toUri().getScheme());
>>>>     [javac]                                ^
>>>>     [javac] 12 errors
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 10:49 AM, Travis Crawford
>>>> <traviscrawford@gmail.com> wrote:
>>>>>
>>>>> On Thu, Jan 24, 2013 at 9:47 AM, Alan Gates <gates@hortonworks.com>
>>>>> wrote:
>>>>>>
>>>>>> If it's not too much trouble the rat excludes file sounds good so
that
>>>>>> every time we release we don't have to remember that these files
are ok
>>>>>> without a header.
>>>>>
>>>>>
>>>>> Should be easy - I'll take a look at this too.
>>>>>
>>>>> --travis
>>>>>
>>>>>
>>>>>>
>>>>>> Alan.
>>>>>>
>>>>>> On Jan 24, 2013, at 9:45 AM, Travis Crawford wrote:
>>>>>>
>>>>>> > On Thu, Jan 24, 2013 at 9:38 AM, Alan Gates <gates@hortonworks.com>
>>>>>> > wrote:
>>>>>> >
>>>>>> > On Jan 24, 2013, at 8:56 AM, Travis Crawford wrote:
>>>>>> >
>>>>>> > > Before simply excluding these files, this is a great opportunity
>>>>>> > > to understand what these files are and potentially fix.
>>>>>> > >
>>>>>> > > (a) The .rej file was simply an error and I will remove.
I believe
>>>>>> > > it was produced when applying HCATALOG-605, which has different
version
>>>>>> > > numbers in trunk/release branches.
>>>>>> > >
>>>>>> > > (b) rcfiles - should these be checked in at all? These
seem like
>>>>>> > > files that should be generated when running tests, not
checked into the
>>>>>> > > source tree. Can someone with more familiarity with the
e2e test data clue
>>>>>> > > us into if generating makes sense?
>>>>>> >
>>>>>> > I agree these should be generated rather than checked in.  Someone
>>>>>> > just needs to write the code to generate rcfiles during the
deploy phase.
>>>>>> > We should file a JIRA on this.  But I don't think we should
delay the
>>>>>> > release over it.
>>>>>> >
>>>>>> > Agreed - I filed https://issues.apache.org/jira/browse/HCATALOG-607
>>>>>> > to fix in the future but not block the release over.
>>>>>> >
>>>>>> > Do we add a rat excludes file, or let these be listed in the
report?
>>>>>> > I'll update the release docs for next time too.
>>>>>> >
>>>>>> >
>>>>>> > >
>>>>>> > > (c) text data files - we could add the header to these
files, and
>>>>>> > > update whatever reads them to ignore comment lines.
>>>>>> >
>>>>>> > There's no need to put headers in these.  Data files are a well
>>>>>> > recognized exception in license headers.
>>>>>> >
>>>>>> > Same question as above - for files that do not need a header,
do we
>>>>>> > let them be listed in the report or explicitly exclude them?
>>>>>> >
>>>>>> >
>>>>>> > >
>>>>>> > > (d) storage handler test includes/excludes - these are
still
>>>>>> > > around because storagehendlers has not been migrated to
the new build
>>>>>> > > templates. Unless there are objections I'd like to remove
these files
>>>>>> > > entirely. If we want to ignore a test in the future it
would need to have an
>>>>>> > > @Ignore annotation.
>>>>>> >
>>>>>> > +1, since excluded-tests is empty.
>>>>>> >
>>>>>> > I'm running tests for
>>>>>> > https://issues.apache.org/jira/browse/HCATALOG-606 now and will
post the
>>>>>> > patch shortly.
>>>>>> >
>>>>>> > --travis
>>>>>> >
>>>>>> >
>>>>>> > >
>>>>>> > > Thoughts?
>>>>>> > >
>>>>>> > > --travis
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > > On Thu, Jan 24, 2013 at 1:08 AM, Sushanth Sowmyan
>>>>>> > > <khorgath@gmail.com> wrote:
>>>>>> > > Also, the release audit lists 7 other unapproved license
:
>>>>>> > >
>>>>>> > > http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/releaseaudit_report.txt
>>>>>> > >
>>>>>> > >
>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/boolean.rcfile
>>>>>> > >
>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/complex.rcfile
>>>>>> > >
>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.rcfile
>>>>>> > >
>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.txt
>>>>>> > >
>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/templeton/inpdir/nums.txt
>>>>>> > >
>>>>>> > > hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/all-tests
>>>>>> > >
>>>>>> > > hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/excluded-tests
>>>>>> > >
>>>>>> > > We need to add these to -e patterns for the releaseaudit
target.
>>>>>> > >
>>>>>> > > -Sushanth
>>>>>> > >
>>>>>> > > On Wed, Jan 23, 2013 at 5:22 PM, Daniel Dai
>>>>>> > > <daijy@hortonworks.com> wrote:
>>>>>> > > > I see a release.sh.rej in the package.
>>>>>> > > >
>>>>>> > > > Thanks,
>>>>>> > > > Daniel
>>>>>> > > >
>>>>>> > > > On Wed, Jan 23, 2013 at 2:03 PM, Travis Crawford
>>>>>> > > > <traviscrawford@gmail.com> wrote:
>>>>>> > > >> Hi,
>>>>>> > > >>
>>>>>> > > >> I have created a candidate build for HCatalog
0.5.0-incubating.
>>>>>> > > >>
>>>>>> > > >> Keys used to sign the release are available at
>>>>>> > > >>
>>>>>> > > >> http://svn.apache.org/viewvc/incubator/hcatalog/trunk/KEYS?view=markup
>>>>>> > > >>
>>>>>> > > >> Please download, test, and try it out:
>>>>>> > > >>
>>>>>> > > >>
>>>>>> > > >> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/
>>>>>> > > >>
>>>>>> > > >> The release, md5 signature, gpg signature, and
rat report can
>>>>>> > > >> all be found
>>>>>> > > >> at the above address.
>>>>>> > > >>
>>>>>> > > >> Should we release this? Vote closes on Monday,
January 28.
>>>>>> > > >>
>>>>>> > > >> --travis
>>>>>> > >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message