hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Jefferson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6336) Issue is hive 12 datanucleus incompatability with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
Date Sat, 01 Feb 2014 13:56:08 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888599#comment-13888599
] 

Andy Jefferson commented on HIVE-6336:
--------------------------------------

For some reason your HIVE pom has "datanucleus-api-jdo" with exclusions for "jdo2-api", "junit",
"log4j" ... yet these are clearly seen to be either non-existent (jdo2-api) or "test" scope
only (junit, log4j) for the "datanucleus-api-jdo" project itself ; such exclusions have no
logical reason to be (IMHO). See http://central.maven.org/maven2/org/datanucleus/datanucleus-api-jdo/3.2.1/datanucleus-api-jdo-3.2.1.pom

> Issue is hive 12 datanucleus incompatability with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-6336
>                 URL: https://issues.apache.org/jira/browse/HIVE-6336
>             Project: Hive
>          Issue Type: Wish
>          Components: HiveServer2
>    Affects Versions: 0.12.0
>         Environment:  Hadoop 2.2  local derby Meatastore embedded
>            Reporter: Nigel Savage
>            Priority: Blocker
>              Labels: HADOOP
>
> There is an with hive 12 datanucleus incompatability which seems to have invompatibility
with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> The main question: 
> *IF hive 0.12.0  and datanucleus are compatabile, then what is the version of datanucleus
I should be using with Hive 12 and Hadoop 2.2?*
> The error which Im getting (this blocks me from properly running hive queries invoked
from the "test" phase of a maven project)
> *To reproduce*
> I have hadoop and hive  running as a pseudo cluster local mode and derby as the metastore
> I have the following environment variables
> {noformat}
> HADOOP_HOME=/home/ubu/hadoop
> JAVA_HOME=/usr/lib/jvm/java-7-oracle
> {noformat}
> I have the RegexSerDe declared in the hive-site.xml
> {noformat}
>     <property>
>         <name>hive.aux.jars.path</name>
>         <value>file:///home/ubu/hadoop/lib/hive-contrib-0.12.0.jar </value>
>         <description>This JAR file  available to all users for alljobs</description>
>     </property>
> {noformat}
> If I run with
> {noformat} 
> <datanucleus.version>3.0.2</datanucleus.version> 
> {noformat}
> I get the following 1 exception only
> 'java.lang.ClassNotFoundException...org.datanucleus.store.types.backed.Ma'  
> HOWEVER, If I run with  
> {noformat}
> <datanucleus.version>3.2.0-release</datanucleus.version> 
> {noformat}
> I get the following 1 exception exception only
> java.lang.ClassNotFoundException:
> org/apache/hadoop/hive/contrib/serde2/RegexSerDe 
> EXPLANATION 
> The RegexSerDe class is picked up at run time but the datanucleus Map class is not available,
I have checked in the datanucleus-core 3.0.2 jar and it is missing,  Upgrading to the first
datanucleus above 3.0.2 that includes the Map class throws the ClassNotFoundException for
RegexSerDe. 
> The earlier *3.0.2* datanucleus, code fails with the missing Map class but the RegexSerDe
class is found, then when I upgrade to the 
> 3.2.0-release the Map class is found but for some unkown reason the code/Hive no longer
finds the RegexSerDe class
> I started using the same datanucleus dependencies found in this hive pom
> http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
> below are the dependencies my latest attempts to get a functioning pom
> {noformat}
>         <dependency>
>             <groupId>org.apache.hbase</groupId>
>             <artifactId>hbase-server</artifactId>
>             <version>0.96.0-hadoop2</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hbase</groupId>
>             <artifactId>hbase-client</artifactId>
>             <version>0.96.0-hadoop2</version>
>         </dependency>
>         <!-- misc -->
>         <dependency>
>             <groupId>org.apache.commons</groupId>
>             <artifactId>commons-lang3</artifactId>
>             <version>3.1</version>
>         </dependency>
>         <dependency>
>             <groupId>com.google.guava</groupId>
>             <artifactId>guava</artifactId>
>             <version>${guava.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.derby</groupId>
>             <artifactId>derby</artifactId>
>             <version>${derby.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.datanucleus</groupId>
>             <artifactId>datanucleus-core</artifactId>
>             <version>${datanucleus.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.datanucleus</groupId>
>             <artifactId>datanucleus-rdbms</artifactId>
>             <version>${datanucleus-rdbms.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>javax.jdo</groupId>
>             <artifactId>jdo-api</artifactId>
>             <version>3.0.1</version>
>         </dependency>
>         <dependency>
>             <groupId>org.datanucleus</groupId>
>             <artifactId>datanucleus-api-jdo</artifactId>
>             <version>${datanucleus.jdo.version}</version>
>             <exclusions>
>                 <exclusion>
>                     <groupId>javax.jdo</groupId>
>                     <artifactId>jdo2-api</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>junit</groupId>
>                     <artifactId>junit</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>log4j</groupId>
>                     <artifactId>log4j</artifactId>
>                 </exclusion>
>             </exclusions>
>         </dependency>
>         <!-- hadoop -->
>         <dependency>
>             <groupId>org.apache.hadoop</groupId>
>             <artifactId>hadoop-client</artifactId>
>             <version>${hadoop.version}</version>
>         </dependency>
>         <!-- hive -->
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-common</artifactId>
>             <version>${hive.version}</version>
>             <scope>provided</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-serde</artifactId>
>             <version>${hive.version}</version>
>             <scope>provided</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-exec</artifactId>
>             <version>${hive.version}</version>
>             <scope>provided</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-jdbc</artifactId>
>             <version>${hive.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-contrib</artifactId>
>             <version>${hive.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-metastore</artifactId>
>             <version>${hive.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-cli</artifactId>
>             <version>${hive.version}</version>
>             <exclusions>
>                 <exclusion>
>                     <groupId>org.datanucleus</groupId>
>                     <artifactId>datanucleus-core</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>org.datanucleus</groupId>
>                     <artifactId>datanucleus-api-jdo</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>org.datanucleus</groupId>
>                     <artifactId>datanucleus-rdbms</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>org.slf4j</groupId>
>                     <artifactId>slf4j-api</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>org.slf4j</groupId>
>                     <artifactId>slf4j-log4j12</artifactId>
>                 </exclusion>
>             </exclusions>
>         </dependency>
>         <dependency>
>             <groupId>com.jolbox</groupId>
>             <artifactId>bonecp</artifactId>
>             <version>${bonecp.version}</version>
>         </dependency>
>         <!-- logging -->
>         <dependency>
>             <groupId>org.slf4j</groupId>
>             <artifactId>slf4j-api</artifactId>
>             <version>${slf4j.version}</version>
>         </dependency>
>         <!-- SL4J Binding provided at runtime -->
>         <dependency>
>             <groupId>log4j</groupId>
>             <artifactId>log4j</artifactId>
>             <version>1.2.12</version>
>             <scope>provided</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.slf4j</groupId>
>             <artifactId>slf4j-log4j12</artifactId>
>             <version>${slf4j.version}</version>
>             <scope>provided</scope>
>         </dependency>
>         <!-- Unit test artifacts -->
>         <dependency>
>             <groupId>junit</groupId>
>             <artifactId>junit</artifactId>
>             <version>4.11</version>
>             <scope>test</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.hamcrest</groupId>
>             <artifactId>hamcrest-all</artifactId>
>             <version>1.3</version>
>             <scope>test</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.mrunit</groupId>
>             <artifactId>mrunit</artifactId>
>             <version>1.0.0</version>
>             <classifier>hadoop2</classifier>
>         </dependency>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message