incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Crawford <traviscrawf...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 01:02:17 GMT
Great to hear! Please let us know if you discover any issues with 0.5.0.

--travis


On Thu, Jan 24, 2013 at 4:39 PM, Timothy Potter <thelabdude@gmail.com>wrote:

> Will try your suggestion, thanks! Btw ... we've been running HCatalog in a
> small-medium sized production Hadoop cluster (~400 cores) for several
> months. It rocks! So I know you guys say it's not ready for production yet,
> but it's been awesome for us.
>
> Cheers,
> Tim
>
>
> On Thu, Jan 24, 2013 at 5:26 PM, Travis Crawford <traviscrawford@gmail.com
> > wrote:
>
>> On Thu, Jan 24, 2013 at 4:20 PM, Timothy Potter <thelabdude@gmail.com>wrote:
>>
>>> Hi Travis,
>>>
>>> Thank you! Using your ant command, I have a build.
>>>
>>
>>  Do you know what changed? You might have had 0.20 hadoop jars already
>> resolved, which would cause a build failure without cleaning.
>>
>>
>>
>>> Are there any instructions or known pitfalls when upgrading from 0.4 to
>>> 0.5?
>>>
>>> Or, even better - can I just replace the 0.4 JARS in
>>> /usr/lib/hive/share/hcatalog with the new 0.5 JARs and restart the
>>> metastore?
>>>
>>
>> The upgrade procedure will differ based on how you have things deployed,
>> primarily if you run the HiveMetaStore as a service or you directly connect
>> to the HMS. You might try running a job with just the HCatalog 0.5 client
>> jars against your existing metastore and see if that works for you. As the
>> vast majority of HCatalog is client-side, its possible to run jobs with
>> different versions while you work out the upgrade procedure for your site.
>>
>> --travis
>>
>>
>>
>>
>>> Cheers,
>>> Tim
>>>
>>> On Thu, Jan 24, 2013 at 5:00 PM, Travis Crawford <
>>> traviscrawford@gmail.com> wrote:
>>>
>>>> Hey Timothy -
>>>>
>>>> On a clean master, this command works for me:
>>>>
>>>>     ant clean package -Dmvn.hadoop.profile=hadoop23
>>>>
>>>> After doing so, we see the following jars were used in the build:
>>>>
>>>> fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
>>>> core/build/lib/compile/hadoop-annotations-0.23.3.jar
>>>> core/build/lib/compile/hadoop-archives-0.23.3.jar
>>>> core/build/lib/compile/hadoop-auth-0.23.3.jar
>>>> core/build/lib/compile/hadoop-common-0.23.3.jar
>>>> core/build/lib/compile/hadoop-hdfs-0.23.3.jar
>>>> core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
>>>> core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
>>>> core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
>>>> fynn-2:hcatalog travis$
>>>>
>>>> I'm not familiar with how CDH4 packages things. Let's see what jars
>>>> you're pulling in first. Also, are you changing this line in the top-level
>>>> pom.xml?
>>>>
>>>> <hadoop23.version>0.23.3</hadoop23.version>
>>>>
>>>> If not, I'm surprised you see issues as you wouldn't even be compiling
>>>> against CDH4.
>>>>
>>>> --travis
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter <thelabdude@gmail.com>wrote:
>>>>
>>>>> Would like to upgrade to 0.5 as we're maintaining 0.4 + many of the
>>>>> patches supplied since the 0.4 release.
>>>>>
>>>>> However, I need HCatalog to run against CDH4.1.2 which based on this
>>>>> link (
>>>>> https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647)
>>>>> is hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+126 ...
>>>>>
>>>>> What is the proper mechanism for building this to run on later
>>>>> versions of Hadoop? The biggest issue is TaskAttemptContext and JobContext
>>>>> are now interfaces in later versions of Hadoop but used to be concrete
Java
>>>>> classes. I think the shims code is trying to insulate us from these issues
>>>>> but when I do:
>>>>>
>>>>> ant -Dmvn.hadoop.profile=hadoop23
>>>>>
>>>>> I get compile errors in the shims (see below).
>>>>>
>>>>> With 0.4 - I hacked up all the build files and some of the code to get
>>>>> a working version. It's looking like I'm going to need to do that again?
>>>>>
>>>>> Here's compile issues when doing the hadoop23 build:
>>>>>
>>>>> jar:
>>>>>      [echo] shims
>>>>>     [javac] Compiling 1 source file to
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/core/build/classes
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:34:
>>>>> package org.apache.hadoop.mapreduce.task does not exist
>>>>>     [javac] import org.apache.hadoop.mapreduce.task.JobContextImpl;
>>>>>     [javac]                                        ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:40:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : class MRJobConfig
>>>>>     [javac] location: package org.apache.hadoop.mapreduce
>>>>>     [javac] import org.apache.hadoop.mapreduce.MRJobConfig;
>>>>>     [javac]                                   ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:46:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : constructor
>>>>> TaskID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int)
>>>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskID
>>>>>     [javac]         return new TaskID("", 0, TaskType.MAP, 0);
>>>>>     [javac]                ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:51:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : constructor
>>>>> TaskAttemptID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int,int)
>>>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskAttemptID
>>>>>     [javac]         return new TaskAttemptID("", 0, TaskType.MAP, 0,
>>>>> 0);
>>>>>     [javac]                ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:57:
>>>>> package org.apache.hadoop.mapreduce.task does not exist
>>>>>     [javac]         return new
>>>>> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(
>>>>>     [javac]                                                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:67:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : class TaskAttemptContextImpl
>>>>>     [javac] location: package org.apache.hadoop.mapred
>>>>>     [javac]             java.lang.reflect.Constructor construct =
>>>>> org.apache.hadoop.mapred.TaskAttemptContextImpl.class.getDeclaredConstructor(
>>>>>     [javac]
>>>>>                     ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:82:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : class JobContextImpl
>>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>>     [javac]         return new JobContextImpl(conf instanceof JobConf?
>>>>> new JobConf(conf) : conf,
>>>>>     [javac]                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:89:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : class JobContextImpl
>>>>>     [javac] location: package org.apache.hadoop.mapred
>>>>>     [javac]         return new org.apache.hadoop.mapred.JobContextImpl(
>>>>>     [javac]                                            ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:114:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : variable MRJobConfig
>>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>>     [javac]             return MRJobConfig.CACHE_ARCHIVES;
>>>>>     [javac]                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:116:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : variable MRJobConfig
>>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>>     [javac]             return MRJobConfig.CACHE_FILES;
>>>>>     [javac]                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:118:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : variable MRJobConfig
>>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>>     [javac]             return MRJobConfig.CACHE_SYMLINK;
>>>>>     [javac]                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:128:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : method resolvePath(org.apache.hadoop.fs.Path)
>>>>>     [javac] location: class org.apache.hadoop.fs.FileSystem
>>>>>     [javac]         return
>>>>> "hdfs".equals(fs.resolvePath(path).toUri().getScheme());
>>>>>     [javac]                                ^
>>>>>     [javac] 12 errors
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 10:49 AM, Travis Crawford <
>>>>> traviscrawford@gmail.com> wrote:
>>>>>
>>>>>> On Thu, Jan 24, 2013 at 9:47 AM, Alan Gates <gates@hortonworks.com>wrote:
>>>>>>
>>>>>>> If it's not too much trouble the rat excludes file sounds good
so
>>>>>>> that every time we release we don't have to remember that these
files are
>>>>>>> ok without a header.
>>>>>>>
>>>>>>
>>>>>> Should be easy - I'll take a look at this too.
>>>>>>
>>>>>> --travis
>>>>>>
>>>>>>
>>>>>>
>>>>>>>  Alan.
>>>>>>>
>>>>>>> On Jan 24, 2013, at 9:45 AM, Travis Crawford wrote:
>>>>>>>
>>>>>>> > On Thu, Jan 24, 2013 at 9:38 AM, Alan Gates <gates@hortonworks.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > On Jan 24, 2013, at 8:56 AM, Travis Crawford wrote:
>>>>>>> >
>>>>>>> > > Before simply excluding these files, this is a great
opportunity
>>>>>>> to understand what these files are and potentially fix.
>>>>>>> > >
>>>>>>> > > (a) The .rej file was simply an error and I will remove.
I
>>>>>>> believe it was produced when applying HCATALOG-605, which has
different
>>>>>>> version numbers in trunk/release branches.
>>>>>>> > >
>>>>>>> > > (b) rcfiles - should these be checked in at all? These
seem like
>>>>>>> files that should be generated when running tests, not checked
into the
>>>>>>> source tree. Can someone with more familiarity with the e2e test
data clue
>>>>>>> us into if generating makes sense?
>>>>>>> >
>>>>>>> > I agree these should be generated rather than checked in.
 Someone
>>>>>>> just needs to write the code to generate rcfiles during the deploy
phase.
>>>>>>>  We should file a JIRA on this.  But I don't think we should
delay the
>>>>>>> release over it.
>>>>>>> >
>>>>>>> > Agreed - I filed
>>>>>>> https://issues.apache.org/jira/browse/HCATALOG-607 to fix in
the
>>>>>>> future but not block the release over.
>>>>>>> >
>>>>>>> > Do we add a rat excludes file, or let these be listed in
the
>>>>>>> report? I'll update the release docs for next time too.
>>>>>>> >
>>>>>>> >
>>>>>>> > >
>>>>>>> > > (c) text data files - we could add the header to these
files,
>>>>>>> and update whatever reads them to ignore comment lines.
>>>>>>> >
>>>>>>> > There's no need to put headers in these.  Data files are
a well
>>>>>>> recognized exception in license headers.
>>>>>>> >
>>>>>>> > Same question as above - for files that do not need a header,
do
>>>>>>> we let them be listed in the report or explicitly exclude them?
>>>>>>> >
>>>>>>> >
>>>>>>> > >
>>>>>>> > > (d) storage handler test includes/excludes - these
are still
>>>>>>> around because storagehendlers has not been migrated to the new
build
>>>>>>> templates. Unless there are objections I'd like to remove these
files
>>>>>>> entirely. If we want to ignore a test in the future it would
need to have
>>>>>>> an @Ignore annotation.
>>>>>>> >
>>>>>>> > +1, since excluded-tests is empty.
>>>>>>> >
>>>>>>> > I'm running tests for
>>>>>>> https://issues.apache.org/jira/browse/HCATALOG-606 now and will
>>>>>>> post the patch shortly.
>>>>>>> >
>>>>>>> > --travis
>>>>>>> >
>>>>>>> >
>>>>>>> > >
>>>>>>> > > Thoughts?
>>>>>>> > >
>>>>>>> > > --travis
>>>>>>> > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > On Thu, Jan 24, 2013 at 1:08 AM, Sushanth Sowmyan <
>>>>>>> khorgath@gmail.com> wrote:
>>>>>>> > > Also, the release audit lists 7 other unapproved license
:
>>>>>>> > >
>>>>>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/releaseaudit_report.txt
>>>>>>> > >
>>>>>>> > >
>>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/boolean.rcfile
>>>>>>> > >
>>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/complex.rcfile
>>>>>>> > >
>>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.rcfile
>>>>>>> > >
>>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.txt
>>>>>>> > >
>>>>>>> hcatalog-src-0.5.0-incubating/src/test/e2e/templeton/inpdir/nums.txt
>>>>>>> > >
>>>>>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/all-tests
>>>>>>> > >
>>>>>>> hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/excluded-tests
>>>>>>> > >
>>>>>>> > > We need to add these to -e patterns for the releaseaudit
target.
>>>>>>> > >
>>>>>>> > > -Sushanth
>>>>>>> > >
>>>>>>> > > On Wed, Jan 23, 2013 at 5:22 PM, Daniel Dai <
>>>>>>> daijy@hortonworks.com> wrote:
>>>>>>> > > > I see a release.sh.rej in the package.
>>>>>>> > > >
>>>>>>> > > > Thanks,
>>>>>>> > > > Daniel
>>>>>>> > > >
>>>>>>> > > > On Wed, Jan 23, 2013 at 2:03 PM, Travis Crawford
>>>>>>> > > > <traviscrawford@gmail.com> wrote:
>>>>>>> > > >> Hi,
>>>>>>> > > >>
>>>>>>> > > >> I have created a candidate build for HCatalog
>>>>>>> 0.5.0-incubating.
>>>>>>> > > >>
>>>>>>> > > >> Keys used to sign the release are available
at
>>>>>>> > > >>
>>>>>>> http://svn.apache.org/viewvc/incubator/hcatalog/trunk/KEYS?view=markup
>>>>>>> > > >>
>>>>>>> > > >> Please download, test, and try it out:
>>>>>>> > > >>
>>>>>>> > > >>
>>>>>>> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/
>>>>>>> > > >>
>>>>>>> > > >> The release, md5 signature, gpg signature,
and rat report can
>>>>>>> all be found
>>>>>>> > > >> at the above address.
>>>>>>> > > >>
>>>>>>> > > >> Should we release this? Vote closes on Monday,
January 28.
>>>>>>> > > >>
>>>>>>> > > >> --travis
>>>>>>> > >
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message