incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sushanth Sowmyan <khorg...@gmail.com>
Subject Re: Building HCatalog 0.5.0-incubating (candidate 1) to run on Cloudera CDH 4.1.2 (hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+1265)
Date Fri, 25 Jan 2013 01:07:06 GMT
And also, in this case, in the core/build/lib/compile/ directory, I
see the following versions of hadoop jars (notice hadoop-core-1.0.3 in
the midst of the others):

--
hadoop-annotations-2.0.2.22-alpha.jar
hadoop-archives-2.0.2.22-alpha.jar
hadoop-auth-2.0.2.22-alpha.jar
hadoop-common-2.0.2.22-alpha.jar
hadoop-core-1.0.3.jar
hadoop-hdfs-2.0.2.22-alpha.jar
hadoop-mapreduce-client-core-2.0.2.22-alpha.jar
hadoop-yarn-api-2.0.2.22-alpha.jar
hadoop-yarn-common-2.0.2.22-alpha.jar
--

On Thu, Jan 24, 2013 at 5:03 PM, Sushanth Sowmyan <khorgath@gmail.com> wrote:
> Interestingly, I'm hitting the same issue with a bigtop build for
> hadoop-2. And this one isn't solved by doing a clean.
>
> On Thu, Jan 24, 2013 at 4:26 PM, Travis Crawford
> <traviscrawford@gmail.com> wrote:
>> On Thu, Jan 24, 2013 at 4:20 PM, Timothy Potter <thelabdude@gmail.com>
>> wrote:
>>>
>>> Hi Travis,
>>>
>>> Thank you! Using your ant command, I have a build.
>>
>>
>> Do you know what changed? You might have had 0.20 hadoop jars already
>> resolved, which would cause a build failure without cleaning.
>>
>>
>>>
>>> Are there any instructions or known pitfalls when upgrading from 0.4 to
>>> 0.5?
>>>
>>> Or, even better - can I just replace the 0.4 JARS in
>>> /usr/lib/hive/share/hcatalog with the new 0.5 JARs and restart the
>>> metastore?
>>
>>
>> The upgrade procedure will differ based on how you have things deployed,
>> primarily if you run the HiveMetaStore as a service or you directly connect
>> to the HMS. You might try running a job with just the HCatalog 0.5 client
>> jars against your existing metastore and see if that works for you. As the
>> vast majority of HCatalog is client-side, its possible to run jobs with
>> different versions while you work out the upgrade procedure for your site.
>>
>> --travis
>>
>>
>>
>>>
>>> Cheers,
>>> Tim
>>>
>>> On Thu, Jan 24, 2013 at 5:00 PM, Travis Crawford
>>> <traviscrawford@gmail.com> wrote:
>>>>
>>>> Hey Timothy -
>>>>
>>>> On a clean master, this command works for me:
>>>>
>>>>     ant clean package -Dmvn.hadoop.profile=hadoop23
>>>>
>>>> After doing so, we see the following jars were used in the build:
>>>>
>>>> fynn-2:hcatalog travis$ ls -1 core/build/lib/compile/*hadoop*
>>>> core/build/lib/compile/hadoop-annotations-0.23.3.jar
>>>> core/build/lib/compile/hadoop-archives-0.23.3.jar
>>>> core/build/lib/compile/hadoop-auth-0.23.3.jar
>>>> core/build/lib/compile/hadoop-common-0.23.3.jar
>>>> core/build/lib/compile/hadoop-hdfs-0.23.3.jar
>>>> core/build/lib/compile/hadoop-mapreduce-client-core-0.23.3.jar
>>>> core/build/lib/compile/hadoop-yarn-api-0.23.3.jar
>>>> core/build/lib/compile/hadoop-yarn-common-0.23.3.jar
>>>> fynn-2:hcatalog travis$
>>>>
>>>> I'm not familiar with how CDH4 packages things. Let's see what jars
>>>> you're pulling in first. Also, are you changing this line in the top-level
>>>> pom.xml?
>>>>
>>>> <hadoop23.version>0.23.3</hadoop23.version>
>>>>
>>>> If not, I'm surprised you see issues as you wouldn't even be compiling
>>>> against CDH4.
>>>>
>>>> --travis
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 3:40 PM, Timothy Potter <thelabdude@gmail.com>
>>>> wrote:
>>>>>
>>>>> Would like to upgrade to 0.5 as we're maintaining 0.4 + many of the
>>>>> patches supplied since the 0.4 release.
>>>>>
>>>>> However, I need HCatalog to run against CDH4.1.2 which based on this
>>>>> link
>>>>> (https://ccp.cloudera.com/display/DOC/CDH+Version+and+Packaging+Information#CDHVersionandPackagingInformation-id511647)
>>>>> is hadoop-2.0.0+552 / hadoop-0.20-mapreduce-0.20.2+126 ...
>>>>>
>>>>> What is the proper mechanism for building this to run on later versions
>>>>> of Hadoop? The biggest issue is TaskAttemptContext and JobContext are
now
>>>>> interfaces in later versions of Hadoop but used to be concrete Java classes.
>>>>> I think the shims code is trying to insulate us from these issues but
when I
>>>>> do:
>>>>>
>>>>> ant -Dmvn.hadoop.profile=hadoop23
>>>>>
>>>>> I get compile errors in the shims (see below).
>>>>>
>>>>> With 0.4 - I hacked up all the build files and some of the code to get
a
>>>>> working version. It's looking like I'm going to need to do that again?
>>>>>
>>>>> Here's compile issues when doing the hadoop23 build:
>>>>>
>>>>> jar:
>>>>>      [echo] shims
>>>>>     [javac] Compiling 1 source file to
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/core/build/classes
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:34:
>>>>> package org.apache.hadoop.mapreduce.task does not exist
>>>>>     [javac] import org.apache.hadoop.mapreduce.task.JobContextImpl;
>>>>>     [javac]                                        ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:40:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : class MRJobConfig
>>>>>     [javac] location: package org.apache.hadoop.mapreduce
>>>>>     [javac] import org.apache.hadoop.mapreduce.MRJobConfig;
>>>>>     [javac]                                   ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:46:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : constructor
>>>>> TaskID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int)
>>>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskID
>>>>>     [javac]         return new TaskID("", 0, TaskType.MAP, 0);
>>>>>     [javac]                ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:51:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : constructor
>>>>> TaskAttemptID(java.lang.String,int,org.apache.hadoop.mapreduce.TaskType,int,int)
>>>>>     [javac] location: class org.apache.hadoop.mapreduce.TaskAttemptID
>>>>>     [javac]         return new TaskAttemptID("", 0, TaskType.MAP, 0,
0);
>>>>>     [javac]                ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:57:
>>>>> package org.apache.hadoop.mapreduce.task does not exist
>>>>>     [javac]         return new
>>>>> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(
>>>>>     [javac]                                                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:67:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : class TaskAttemptContextImpl
>>>>>     [javac] location: package org.apache.hadoop.mapred
>>>>>     [javac]             java.lang.reflect.Constructor construct =
>>>>> org.apache.hadoop.mapred.TaskAttemptContextImpl.class.getDeclaredConstructor(
>>>>>     [javac]
>>>>> ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:82:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : class JobContextImpl
>>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>>     [javac]         return new JobContextImpl(conf instanceof JobConf?
>>>>> new JobConf(conf) : conf,
>>>>>     [javac]                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:89:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : class JobContextImpl
>>>>>     [javac] location: package org.apache.hadoop.mapred
>>>>>     [javac]         return new org.apache.hadoop.mapred.JobContextImpl(
>>>>>     [javac]                                            ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:114:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : variable MRJobConfig
>>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>>     [javac]             return MRJobConfig.CACHE_ARCHIVES;
>>>>>     [javac]                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:116:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : variable MRJobConfig
>>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>>     [javac]             return MRJobConfig.CACHE_FILES;
>>>>>     [javac]                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:118:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : variable MRJobConfig
>>>>>     [javac] location: class org.apache.hcatalog.shims.HCatHadoopShims23
>>>>>     [javac]             return MRJobConfig.CACHE_SYMLINK;
>>>>>     [javac]                    ^
>>>>>     [javac]
>>>>> /Users/timpotter/dev/hcat-0.5/hcatalog-src-0.5.0-incubating/shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java:128:
>>>>> cannot find symbol
>>>>>     [javac] symbol  : method resolvePath(org.apache.hadoop.fs.Path)
>>>>>     [javac] location: class org.apache.hadoop.fs.FileSystem
>>>>>     [javac]         return
>>>>> "hdfs".equals(fs.resolvePath(path).toUri().getScheme());
>>>>>     [javac]                                ^
>>>>>     [javac] 12 errors
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 10:49 AM, Travis Crawford
>>>>> <traviscrawford@gmail.com> wrote:
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 9:47 AM, Alan Gates <gates@hortonworks.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> If it's not too much trouble the rat excludes file sounds good
so that
>>>>>>> every time we release we don't have to remember that these files
are ok
>>>>>>> without a header.
>>>>>>
>>>>>>
>>>>>> Should be easy - I'll take a look at this too.
>>>>>>
>>>>>> --travis
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Alan.
>>>>>>>
>>>>>>> On Jan 24, 2013, at 9:45 AM, Travis Crawford wrote:
>>>>>>>
>>>>>>> > On Thu, Jan 24, 2013 at 9:38 AM, Alan Gates <gates@hortonworks.com>
>>>>>>> > wrote:
>>>>>>> >
>>>>>>> > On Jan 24, 2013, at 8:56 AM, Travis Crawford wrote:
>>>>>>> >
>>>>>>> > > Before simply excluding these files, this is a great
opportunity
>>>>>>> > > to understand what these files are and potentially
fix.
>>>>>>> > >
>>>>>>> > > (a) The .rej file was simply an error and I will remove.
I believe
>>>>>>> > > it was produced when applying HCATALOG-605, which has
different version
>>>>>>> > > numbers in trunk/release branches.
>>>>>>> > >
>>>>>>> > > (b) rcfiles - should these be checked in at all? These
seem like
>>>>>>> > > files that should be generated when running tests,
not checked into the
>>>>>>> > > source tree. Can someone with more familiarity with
the e2e test data clue
>>>>>>> > > us into if generating makes sense?
>>>>>>> >
>>>>>>> > I agree these should be generated rather than checked in.
 Someone
>>>>>>> > just needs to write the code to generate rcfiles during
the deploy phase.
>>>>>>> > We should file a JIRA on this.  But I don't think we should
delay the
>>>>>>> > release over it.
>>>>>>> >
>>>>>>> > Agreed - I filed https://issues.apache.org/jira/browse/HCATALOG-607
>>>>>>> > to fix in the future but not block the release over.
>>>>>>> >
>>>>>>> > Do we add a rat excludes file, or let these be listed in
the report?
>>>>>>> > I'll update the release docs for next time too.
>>>>>>> >
>>>>>>> >
>>>>>>> > >
>>>>>>> > > (c) text data files - we could add the header to these
files, and
>>>>>>> > > update whatever reads them to ignore comment lines.
>>>>>>> >
>>>>>>> > There's no need to put headers in these.  Data files are
a well
>>>>>>> > recognized exception in license headers.
>>>>>>> >
>>>>>>> > Same question as above - for files that do not need a header,
do we
>>>>>>> > let them be listed in the report or explicitly exclude them?
>>>>>>> >
>>>>>>> >
>>>>>>> > >
>>>>>>> > > (d) storage handler test includes/excludes - these
are still
>>>>>>> > > around because storagehendlers has not been migrated
to the new build
>>>>>>> > > templates. Unless there are objections I'd like to
remove these files
>>>>>>> > > entirely. If we want to ignore a test in the future
it would need to have an
>>>>>>> > > @Ignore annotation.
>>>>>>> >
>>>>>>> > +1, since excluded-tests is empty.
>>>>>>> >
>>>>>>> > I'm running tests for
>>>>>>> > https://issues.apache.org/jira/browse/HCATALOG-606 now and
will post the
>>>>>>> > patch shortly.
>>>>>>> >
>>>>>>> > --travis
>>>>>>> >
>>>>>>> >
>>>>>>> > >
>>>>>>> > > Thoughts?
>>>>>>> > >
>>>>>>> > > --travis
>>>>>>> > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > On Thu, Jan 24, 2013 at 1:08 AM, Sushanth Sowmyan
>>>>>>> > > <khorgath@gmail.com> wrote:
>>>>>>> > > Also, the release audit lists 7 other unapproved license
:
>>>>>>> > >
>>>>>>> > > http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/releaseaudit_report.txt
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/boolean.rcfile
>>>>>>> > >
>>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/complex.rcfile
>>>>>>> > >
>>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.rcfile
>>>>>>> > >
>>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/hcatalog/data/numbers.txt
>>>>>>> > >
>>>>>>> > > hcatalog-src-0.5.0-incubating/src/test/e2e/templeton/inpdir/nums.txt
>>>>>>> > >
>>>>>>> > > hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/all-tests
>>>>>>> > >
>>>>>>> > > hcatalog-src-0.5.0-incubating/storage-handlers/hbase/src/test/excluded-tests
>>>>>>> > >
>>>>>>> > > We need to add these to -e patterns for the releaseaudit
target.
>>>>>>> > >
>>>>>>> > > -Sushanth
>>>>>>> > >
>>>>>>> > > On Wed, Jan 23, 2013 at 5:22 PM, Daniel Dai
>>>>>>> > > <daijy@hortonworks.com> wrote:
>>>>>>> > > > I see a release.sh.rej in the package.
>>>>>>> > > >
>>>>>>> > > > Thanks,
>>>>>>> > > > Daniel
>>>>>>> > > >
>>>>>>> > > > On Wed, Jan 23, 2013 at 2:03 PM, Travis Crawford
>>>>>>> > > > <traviscrawford@gmail.com> wrote:
>>>>>>> > > >> Hi,
>>>>>>> > > >>
>>>>>>> > > >> I have created a candidate build for HCatalog
0.5.0-incubating.
>>>>>>> > > >>
>>>>>>> > > >> Keys used to sign the release are available
at
>>>>>>> > > >>
>>>>>>> > > >> http://svn.apache.org/viewvc/incubator/hcatalog/trunk/KEYS?view=markup
>>>>>>> > > >>
>>>>>>> > > >> Please download, test, and try it out:
>>>>>>> > > >>
>>>>>>> > > >>
>>>>>>> > > >> http://people.apache.org/~travis/hcatalog-0.5.0-incubating-candidate-1/
>>>>>>> > > >>
>>>>>>> > > >> The release, md5 signature, gpg signature,
and rat report can
>>>>>>> > > >> all be found
>>>>>>> > > >> at the above address.
>>>>>>> > > >>
>>>>>>> > > >> Should we release this? Vote closes on Monday,
January 28.
>>>>>>> > > >>
>>>>>>> > > >> --travis
>>>>>>> > >
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Mime
View raw message