pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Jarcec Cecho (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3285) Jobs using HBaseStorage fail to ship dependency jars
Date Tue, 19 Nov 2013 23:23:27 GMT

    [ https://issues.apache.org/jira/browse/PIG-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827083#comment-13827083
] 

Jarek Jarcec Cecho commented on PIG-3285:
-----------------------------------------

Thank you for the quick turnaround [~ndimiduk]!

I was able to run {{TestHBaseStorage}} unit tests for all the combinations using following
commands:

{code}
# Old path:
ant clean test -Dhbaseversion=94 -Dtestcase=TestHBaseStorage
ant clean test -Dhbaseversion=95 -Dtestcase=TestHBaseStorage  -Dprotobuf-java.version=2.5.0

# New path:

# Compile HBase 0.94 branch and install it to local maven cache (mvn clean install -DskipTests)
ant clean test -Dhbaseversion=94 -Dtestcase=TestHBaseStorage -Dhbase94.version=0.94.14
# Compile HBase 0.96 branch and install it to local maven cache (mvn clean install -DskipTests)
ant clean test -Dhbaseversion=95 -Dtestcase=TestHBaseStorage  -Dprotobuf-java.version=2.5.0
-Dhbase95.version=0.96.1-SNAPSHOT
{code}

The only problem that I've encounter so far is the exception {{ReflectiveOperationException}}
([javadoc|http://docs.oracle.com/javase/7/docs/api/java/lang/ReflectiveOperationException.html])
that has been added in JDK7, while Pig still requires JDK6 compatibility. I've substituted
the exception with {{Exception}}  for the purpose of above tests.

I'll try to give it a spin on a real clusters to see if everything will work as expected.


> Jobs using HBaseStorage fail to ship dependency jars
> ----------------------------------------------------
>
>                 Key: PIG-3285
>                 URL: https://issues.apache.org/jira/browse/PIG-3285
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>             Fix For: 0.13.0
>
>         Attachments: 0001-PIG-3285-Add-HBase-dependency-jars.patch, 0001-PIG-3285-Add-HBase-dependency-jars.patch,
0001-PIG-3285-Add-HBase-dependency-jars.patch, 0001-PIG-3285-Add-HBase-dependency-jars.patch,
0001-PIG-3285-Add-HBase-dependency-jars.patch, 1.pig, 1.txt, 2.pig
>
>
> Launching a job consuming {{HBaseStorage}} fails out of the box. The user must specify
{{-Dpig.additional.jars}} for HBase and all of its dependencies. Exceptions look something
like this:
> {noformat}
> 2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoClassDefFoundError:
com/google/protobuf/Message
> 	at org.apache.hadoop.hbase.io.HbaseObjectWritable.<clinit>(HbaseObjectWritable.java:266)
> 	at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
> 	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
> 	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
> 	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
> 	at $Proxy7.getProtocolVersion(Unknown Source)
> 	at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
> 	at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message