hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nigel Savage (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6336) Issue is hive 12 datanucleus incompatability with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
Date Sun, 02 Feb 2014 00:14:09 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888781#comment-13888781
] 

Nigel Savage commented on HIVE-6336:
------------------------------------

Thanks andy for your response

you are correct re datanucleus-api-jdo exclusions, thanks for the link to the pom, the reason
the exclusions are there is because i used the org.apache.hive:hive-metastore:0.12.0 pom as
a guide, I assumed that this pom would be correct, that is if the exclusions are there then
you need them seeing we use this class in our code and it is the main database that is jdo
in terms of transactions
http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom

basically I am getting the exceptions outlined and I am trying to resolve the issue so I am
trying diferent things, basically trying to harmonize the pom so it uses the same datanucleus
as the dependencies like the org.apache.hive:hive-metastore:0.12.0

I notice you have a datanucleus email so if possible would you be willing to comment on what
versions of datanucleus we should be using ?
for hive 12 and hadoop 2.2.0
<datanucleus.version>3.2.2</datanucleus.version> for  <artifactId>datanucleus-core</artifactId>
<datanucleus-rdbms.version>3.2.1</datanucleus-rdbms.version> <artifactId>datanucleus-rdbms</artifactId>
<datanucleus.jdo.version>3.2.1</datanucleus.jdo.version> <artifactId>datanucleus-api-jdo</artifactId>

> Issue is hive 12 datanucleus incompatability with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-6336
>                 URL: https://issues.apache.org/jira/browse/HIVE-6336
>             Project: Hive
>          Issue Type: Wish
>          Components: HiveServer2
>    Affects Versions: 0.12.0
>         Environment:  Hadoop 2.2  local derby Meatastore embedded
>            Reporter: Nigel Savage
>            Priority: Blocker
>              Labels: HADOOP
>
> There is an with hive 12 datanucleus incompatability which seems to have invompatibility
with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> The main question: 
> *IF hive 0.12.0  and datanucleus are compatabile, then what is the version of datanucleus
I should be using with Hive 12 and Hadoop 2.2?*
> The error which Im getting (this blocks me from properly running hive queries invoked
from the "test" phase of a maven project)
> *To reproduce*
> I have hadoop and hive  running as a pseudo cluster local mode and derby as the metastore
> I have the following environment variables
> {noformat}
> HADOOP_HOME=/home/ubu/hadoop
> JAVA_HOME=/usr/lib/jvm/java-7-oracle
> {noformat}
> I have the RegexSerDe declared in the hive-site.xml
> {noformat}
>     <property>
>         <name>hive.aux.jars.path</name>
>         <value>file:///home/ubu/hadoop/lib/hive-contrib-0.12.0.jar </value>
>         <description>This JAR file  available to all users for alljobs</description>
>     </property>
> {noformat}
> If I run with
> {noformat} 
> <datanucleus.version>3.0.2</datanucleus.version> 
> {noformat}
> I get the following 1 exception only
> 'java.lang.ClassNotFoundException...org.datanucleus.store.types.backed.Ma'  
> HOWEVER, If I run with  
> {noformat}
> <datanucleus.version>3.2.0-release</datanucleus.version> 
> {noformat}
> I get the following 1 exception exception only
> java.lang.ClassNotFoundException:
> org/apache/hadoop/hive/contrib/serde2/RegexSerDe 
> EXPLANATION 
> The RegexSerDe class is picked up at run time but the datanucleus Map class is not available,
I have checked in the datanucleus-core 3.0.2 jar and it is missing,  Upgrading to the first
datanucleus above 3.0.2 that includes the Map class throws the ClassNotFoundException for
RegexSerDe. 
> The earlier *3.0.2* datanucleus, code fails with the missing Map class but the RegexSerDe
class is found, then when I upgrade to the 
> 3.2.0-release the Map class is found but for some unkown reason the code/Hive no longer
finds the RegexSerDe class
> I started using the same datanucleus dependencies found in this hive pom
> http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
> below are the dependencies my latest attempts to get a functioning pom
> {noformat}
>         <dependency>
>             <groupId>org.apache.hbase</groupId>
>             <artifactId>hbase-server</artifactId>
>             <version>0.96.0-hadoop2</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hbase</groupId>
>             <artifactId>hbase-client</artifactId>
>             <version>0.96.0-hadoop2</version>
>         </dependency>
>         <!-- misc -->
>         <dependency>
>             <groupId>org.apache.commons</groupId>
>             <artifactId>commons-lang3</artifactId>
>             <version>3.1</version>
>         </dependency>
>         <dependency>
>             <groupId>com.google.guava</groupId>
>             <artifactId>guava</artifactId>
>             <version>${guava.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.derby</groupId>
>             <artifactId>derby</artifactId>
>             <version>${derby.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.datanucleus</groupId>
>             <artifactId>datanucleus-core</artifactId>
>             <version>${datanucleus.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.datanucleus</groupId>
>             <artifactId>datanucleus-rdbms</artifactId>
>             <version>${datanucleus-rdbms.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>javax.jdo</groupId>
>             <artifactId>jdo-api</artifactId>
>             <version>3.0.1</version>
>         </dependency>
>         <dependency>
>             <groupId>org.datanucleus</groupId>
>             <artifactId>datanucleus-api-jdo</artifactId>
>             <version>${datanucleus.jdo.version}</version>
>             <exclusions>
>                 <exclusion>
>                     <groupId>javax.jdo</groupId>
>                     <artifactId>jdo2-api</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>junit</groupId>
>                     <artifactId>junit</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>log4j</groupId>
>                     <artifactId>log4j</artifactId>
>                 </exclusion>
>             </exclusions>
>         </dependency>
>         <!-- hadoop -->
>         <dependency>
>             <groupId>org.apache.hadoop</groupId>
>             <artifactId>hadoop-client</artifactId>
>             <version>${hadoop.version}</version>
>         </dependency>
>         <!-- hive -->
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-common</artifactId>
>             <version>${hive.version}</version>
>             <scope>provided</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-serde</artifactId>
>             <version>${hive.version}</version>
>             <scope>provided</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-exec</artifactId>
>             <version>${hive.version}</version>
>             <scope>provided</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-jdbc</artifactId>
>             <version>${hive.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-contrib</artifactId>
>             <version>${hive.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-metastore</artifactId>
>             <version>${hive.version}</version>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.hive</groupId>
>             <artifactId>hive-cli</artifactId>
>             <version>${hive.version}</version>
>             <exclusions>
>                 <exclusion>
>                     <groupId>org.datanucleus</groupId>
>                     <artifactId>datanucleus-core</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>org.datanucleus</groupId>
>                     <artifactId>datanucleus-api-jdo</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>org.datanucleus</groupId>
>                     <artifactId>datanucleus-rdbms</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>org.slf4j</groupId>
>                     <artifactId>slf4j-api</artifactId>
>                 </exclusion>
>                 <exclusion>
>                     <groupId>org.slf4j</groupId>
>                     <artifactId>slf4j-log4j12</artifactId>
>                 </exclusion>
>             </exclusions>
>         </dependency>
>         <dependency>
>             <groupId>com.jolbox</groupId>
>             <artifactId>bonecp</artifactId>
>             <version>${bonecp.version}</version>
>         </dependency>
>         <!-- logging -->
>         <dependency>
>             <groupId>org.slf4j</groupId>
>             <artifactId>slf4j-api</artifactId>
>             <version>${slf4j.version}</version>
>         </dependency>
>         <!-- SL4J Binding provided at runtime -->
>         <dependency>
>             <groupId>log4j</groupId>
>             <artifactId>log4j</artifactId>
>             <version>1.2.12</version>
>             <scope>provided</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.slf4j</groupId>
>             <artifactId>slf4j-log4j12</artifactId>
>             <version>${slf4j.version}</version>
>             <scope>provided</scope>
>         </dependency>
>         <!-- Unit test artifacts -->
>         <dependency>
>             <groupId>junit</groupId>
>             <artifactId>junit</artifactId>
>             <version>4.11</version>
>             <scope>test</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.hamcrest</groupId>
>             <artifactId>hamcrest-all</artifactId>
>             <version>1.3</version>
>             <scope>test</scope>
>         </dependency>
>         <dependency>
>             <groupId>org.apache.mrunit</groupId>
>             <artifactId>mrunit</artifactId>
>             <version>1.0.0</version>
>             <classifier>hadoop2</classifier>
>         </dependency>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message