incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Crawford <traviscrawf...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 00:26:41 GMT
On Thu, Jan 24, 2013 at 4:20 PM, Timothy Potter <thelabdude@gmail.com>wrote:

> Hi Travis,
>
> Thank you! Using your ant command, I have a build.
>

Do you know what changed? You might have had 0.20 hadoop jars already
resolved, which would cause a build failure without cleaning.



> Are there any instructions or known pitfalls when upgrading from 0.4 to
> 0.5?
>
> Or, even better - can I just replace the 0.4 JARS in
> /usr/lib/hive/share/hcatalog with the new 0.5 JARs and restart the
> metastore?
>

The upgrade procedure will differ based on how you have things deployed,
primarily if you run the HiveMetaStore as a service or you directly connect
to the HMS. You might try running a job with just the HCatalog 0.5 client
jars against your existing metastore and see if that works for you. As the
vast majority of HCatalog is client-side, its possible to run jobs with
different versions while you work out the upgrade procedure for your site.

--travis




> Cheers,
> Tim
>
> On Thu, Jan 24, 2013 at 5:00 PM, Travis Crawford <traviscrawford@gmail.com
> > wrote:
>
>> Hey Timothy -
>>
>> On a clean master, this command works for me:
>>
>>     ant clean package -Dmvn.hadoop.profile=hadoop23
>>
>> After doing so, we see the following jars were used in the build:
>>
>> fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
>> core/build/lib/compile/hadoop-annotations-0.23.3.jar
>> core/build/lib/compile/hadoop-archives-0.23.3.jar
>> core/build/lib/compile/hadoop-auth-0.23.3.jar
>> core/build/lib/compile/hadoop-common-0.23.3.jar
>> core/build/lib/compile/hadoop-hdfs-0.23.3.jar
>> core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
>> core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
>> core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
>> fynn-2:hcatalog travis$
>>
>> I'm not familiar with how CDH4 packages things. Let's see what jars
>> you're pulling in first. Also, are you changing this line in the top-level
>> pom.xml?
>>
>> <hadoop23.version>0.23.3</hadoop23.version>
>>
>> If not, I'm surprised you see issues as you wouldn't even be compiling
>> against CDH4.
>>
>> --travis
>>
>>
>>
>>
>> On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter <thelabdude@gmail.com>wrote:
>>
>>> Would like to upgrade to 0.5 as we're maintaining 0.4 + many of the
>>> patches supplied since the 0.4 release.
>>>
>>> However, I need HCatalog to run against CDH4.1.2 which based on this
>>> link (
>>> https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647)
>>> is hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+126 ...
>>>
>>> What is the proper mechanism for building this to run on later versions
>>> of Hadoop? The biggest issue is TaskAttemptContext and JobContext are now
>>> interfaces in later versions of Hadoop but used to be concrete Java
>>> classes. I think the shims code is trying to insulate us from these issues
>>> but when I do:
>>>
>>> ant -Dmvn.hadoop.profile=hadoop23
>>>
>>> I get compile errors in the shims (see below).
>>>
>>> With 0.4 - I hacked up all the build files and some of the code to get a
>>> working version. It's looking like I'm going to need to do that again?
>>>
>>> Here's compile issues when doing the hadoop23 build:
>>>
>>> jar:
>>>      [echo] shims
>>>     [javac] Compiling 1 source file to
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/core/build/classes
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:34:
>>> package org.apache.hadoop.mapreduce.task does not exist
>>>     [javac] import org.apache.hadoop.mapreduce.task.JobContextImpl;
>>>     [javac]                                        ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:40:
>>> cannot find symbol
>>>     [javac] symbol  : class MRJobConfig
>>>     [javac] location: package org.apache.hadoop.mapreduce
>>>     [javac] import org.apache.hadoop.mapreduce.MRJobConfig;
>>>     [javac]                                   ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:46:
>>> cannot find symbol
>>>     [javac] symbol  : constructor
>>> TaskID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int)
>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskID
>>>     [javac]         return new TaskID("", 0, TaskType.MAP, 0);
>>>     [javac]                ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:51:
>>> cannot find symbol
>>>     [javac] symbol  : constructor
>>> TaskAttemptID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int,int)
>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskAttemptID
>>>     [javac]         return new TaskAttemptID("", 0, TaskType.MAP, 0, 0);
>>>     [javac]                ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:57:
>>> package org.apache.hadoop.mapreduce.task does not exist
>>>     [javac]         return new
>>> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(
>>>     [javac]                                                    ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:67:
>>> cannot find symbol
>>>     [javac] symbol  : class TaskAttemptContextImpl
>>>     [javac] location: package org.apache.hadoop.mapred
>>>     [javac]             java.lang.reflect.Constructor construct =
>>> org.apache.hadoop.mapred.TaskAttemptContextImpl.class.getDeclaredConstructor(
>>>     [javac]
>>>                   ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:82:
>>> cannot find symbol
>>>     [javac] symbol  : class JobContextImpl
>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>     [javac]         return new JobContextImpl(conf instanceof JobConf?
>>> new JobConf(conf) : conf,
>>>     [javac]                    ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:89:
>>> cannot find symbol
>>>     [javac] symbol  : class JobContextImpl
>>>     [javac] location: package org.apache.hadoop.mapred
>>>     [javac]         return new org.apache.hadoop.mapred.JobContextImpl(
>>>     [javac]                                            ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:114:
>>> cannot find symbol
>>>     [javac] symbol  : variable MRJobConfig
>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>     [javac]             return MRJobConfig.CACHE_ARCHIVES;
>>>     [javac]                    ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:116:
>>> cannot find symbol
>>>     [javac] symbol  : variable MRJobConfig
>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>     [javac]             return MRJobConfig.CACHE_FILES;
>>>     [javac]                    ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:118:
>>> cannot find symbol
>>>     [javac] symbol  : variable MRJobConfig
>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>     [javac]             return MRJobConfig.CACHE_SYMLINK;
>>>     [javac]                    ^
>>>     [javac]
>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:128:
>>> cannot find symbol
>>>     [javac] symbol  : method resolvePath(org.apache.hadoop.fs.Path)
>>>     [javac] location: class org.apache.hadoop.fs.FileSystem
>>>     [javac]         return
>>> "hdfs".equals(fs.resolvePath(path).toUri().getScheme());
>>>     [javac]                                ^
>>>     [javac] 12 errors
>>>
>>>
>>> On Thu, Jan 24, 2013 at 10:49 AM, Travis Crawford <
>>> traviscrawford@gmail.com> wrote:
>>>
>>>> On Thu, Jan 24, 2013 at 9:47 AM, Alan Gates <gates@hortonworks.com>wrote:
>>>>
>>>>> If it's not too much trouble the rat excludes file sounds good so that
>>>>> every time we release we don't have to remember that these files are
ok
>>>>> without a header.
>>>>>
>>>>
>>>> Should be easy - I'll take a look at this too.
>>>>
>>>> --travis
>>>>
>>>>
>>>>
>>>>>  Alan.
>>>>>
>>>>> On Jan 24, 2013, at 9:45 AM, Travis Crawford wrote:
>>>>>
>>>>> > On Thu, Jan 24, 2013 at 9:38 AM, Alan Gates <gates@hortonworks.com>
>>>>> wrote:
>>>>> >
>>>>> > On Jan 24, 2013, at 8:56 AM, Travis Crawford wrote:
>>>>> >
>>>>> > > Before simply excluding these files, this is a great opportunity
>>>>> to understand what these files are and potentially fix.
>>>>> > >
>>>>> > > (a) The .rej file was simply an error and I will remove. I
believe
>>>>> it was produced when applying HCATALOG-605, which has different version
>>>>> numbers in trunk/release branches.
>>>>> > >
>>>>> > > (b) rcfiles - should these be checked in at all? These seem
like
>>>>> files that should be generated when running tests, not checked into the
>>>>> source tree. Can someone with more familiarity with the e2e test data
clue
>>>>> us into if generating makes sense?
>>>>> >
>>>>> > I agree these should be generated rather than checked in.  Someone
>>>>> just needs to write the code to generate rcfiles during the deploy phase.
>>>>>  We should file a JIRA on this.  But I don't think we should delay the
>>>>> release over it.
>>>>> >
>>>>> > Agreed - I filed https://issues.apache.org/jira/browse/HCATALOG-607to
fix in the future but not block the release over.
>>>>> >
>>>>> > Do we add a rat excludes file, or let these be listed in the report?
>>>>> I'll update the release docs for next time too.
>>>>> >
>>>>> >
>>>>> > >
>>>>> > > (c) text data files - we could add the header to these files,
and
>>>>> update whatever reads them to ignore comment lines.
>>>>> >
>>>>> > There's no need to put headers in these.  Data files are a well
>>>>> recognized exception in license headers.
>>>>> >
>>>>> > Same question as above - for files that do not need a header, do
we
>>>>> let them be listed in the report or explicitly exclude them?
>>>>> >
>>>>> >
>>>>> > >
>>>>> > > (d) storage handler test includes/excludes - these are still
>>>>> around because storagehendlers has not been migrated to the new build
>>>>> templates. Unless there are objections I'd like to remove these files
>>>>> entirely. If we want to ignore a test in the future it would need to
have
>>>>> an @Ignore annotation.
>>>>> >
>>>>> > +1, since excluded-tests is empty.
>>>>> >
>>>>> > I'm running tests for
>>>>> https://issues.apache.org/jira/browse/HCATALOG-606 now and will post
>>>>> the patch shortly.
>>>>> >
>>>>> > --travis
>>>>> >
>>>>> >
>>>>> > >
>>>>> > > Thoughts?
>>>>> > >
>>>>> > > --travis
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > On Thu, Jan 24, 2013 at 1:08 AM, Sushanth Sowmyan <
>>>>> khorgath@gmail.com> wrote:
>>>>> > > Also, the release audit lists 7 other unapproved license :
>>>>> > >
>>>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/releaseaudit_report.txt
>>>>> > >
>>>>> > >
>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/boolean.rcfile
>>>>> > >
>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/complex.rcfile
>>>>> > >
>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.rcfile
>>>>> > >
>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.txt
>>>>> > >
>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/templeton/inpdir/nums.txt
>>>>> > >
>>>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/all-tests
>>>>> > >
>>>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/excluded-tests
>>>>> > >
>>>>> > > We need to add these to -e patterns for the releaseaudit target.
>>>>> > >
>>>>> > > -Sushanth
>>>>> > >
>>>>> > > On Wed, Jan 23, 2013 at 5:22 PM, Daniel Dai <daijy@hortonworks.com>
>>>>> wrote:
>>>>> > > > I see a release.sh.rej in the package.
>>>>> > > >
>>>>> > > > Thanks,
>>>>> > > > Daniel
>>>>> > > >
>>>>> > > > On Wed, Jan 23, 2013 at 2:03 PM, Travis Crawford
>>>>> > > > <traviscrawford@gmail.com> wrote:
>>>>> > > >> Hi,
>>>>> > > >>
>>>>> > > >> I have created a candidate build for HCatalog 0.5.0-incubating.
>>>>> > > >>
>>>>> > > >> Keys used to sign the release are available at
>>>>> > > >>
>>>>> http://svn.apache.org/viewvc/incubator/hcatalog/trunk/KEYS?view=markup
>>>>> > > >>
>>>>> > > >> Please download, test, and try it out:
>>>>> > > >>
>>>>> > > >>
>>>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/
>>>>> > > >>
>>>>> > > >> The release, md5 signature, gpg signature, and rat
report can
>>>>> all be found
>>>>> > > >> at the above address.
>>>>> > > >>
>>>>> > > >> Should we release this? Vote closes on Monday, January
28.
>>>>> > > >>
>>>>> > > >> --travis
>>>>> > >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message