incubator-crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Build Inconsistencies with Profiles
Date Wed, 23 Jan 2013 16:05:25 GMT
On Wed, Jan 23, 2013 at 5:14 AM, Whitacre,Micah
<MICAH.WHITACRE@cerner.com>wrote:

>  Thanks.  I no longer get that failure when doing the build with
> "-Dcrunch.platform=2".  I'm still getting the failure on my CDH4 fork and
> will keep looking.  Your change gave me an idea of where to look.
>
>  I still do not get a completely successful build for "hadoop-2":
>
>  <testcase time="10.516"
> classname="org.apache.crunch.io.hbase.WordCountHBaseIT"
> name="testWordCount">
>     <error
> message="org.apache.hadoop.hdfs.MiniDFSCluster.&lt;init&gt;(ILorg/apache/hadoop/conf/Configuration;IZZZLorg/apache/hadoop/hdfs/server/common/HdfsConstants$StartupOption;[Ljava/lang/String;[Ljava/lang/String;[J)V"
> type="java.lang.NoSuchMethodError">java.lang.NoSuchMethodError:
> org.apache.hadoop.hdfs.MiniDFSCluster.&lt;init&gt;(ILorg/apache/hadoop/conf/Configuration;IZZZLorg/apache/hadoop/hdfs/server/common/HdfsConstants$StartupOption;[Ljava/lang/String;[Ljava/lang/String;[J)V
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:430)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:598)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:554)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:523)
> at
> org.apache.crunch.io.hbase.WordCountHBaseIT.setUp(WordCountHBaseIT.java:160)
>
>  This actually seems to indicate a dependency mismatch between hbase and
> hadoop-minicluster.  There is a note in the POM about needing to build
> hbase 0.94.1 from src.  Is that still necessary as a released version is
> available in Maven Central[1].
>

Yes, it is. Like Crunch, HBase builds its maven jars against Hadoop 1.x, so
running the Hadoop 2.x unit tests requires a custom build of hbase from
source. That's one of the things bigtop does when they put together one of
their package releases.


>
>  [1] -
> http://search.maven.org/#artifactdetails%7Corg.apache.hbase%7Chbase%7C0.94.1%7Cjar
>
>  On Jan 22, 2013, at 5:53 PM, Josh Wills wrote:
>
>  I think I have a fix, tracking it at:
> https://issues.apache.org/jira/browse/CRUNCH-148<https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/browse/CRUNCH-148&k=PmKqfXspAHNo6iYJ48Q45A%3D%3D%0A&r=MwP8zm6sgnnstbiUpAReMZvSqrZXwpejyuwyb6GLlpU%3D%0A&m=OpOZQK3F8FlJaWn6bDsTi1i0sQeqftvdvxB%2BA5kVfzU%3D%0A&s=90d2b9078d65a5f89f5ed618a2a340fc3db5d1dda006d0e8325309a614f3c3c5>
>
>
>  CONFIDENTIALITY NOTICE This message and any included attachments are from
> Cerner Corporation and are intended only for the addressee. The information
> contained in this message is confidential and may constitute inside or
> non-public information under international, federal, or state securities
> laws. Unauthorized forwarding, printing, copying, distribution, or use of
> such information is strictly prohibited and may be unlawful. If you are not
> the addressee, please promptly delete this message and notify the sender of
> the delivery error by e-mail or you may call Cerner's corporate offices in
> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message