hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Capriolo (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1157) UDFs can't be loaded via "add jar" when jar is on HDFS
Date Mon, 22 Mar 2010 18:17:27 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848261#action_12848261
] 

Edward Capriolo commented on HIVE-1157:
---------------------------------------


{noformat}
[edward@ec hive]$ ant -Dtestcase=TestAddJarFromHDFS test

<testcase classname="org.apache.hadoop.hive.ql.session.TestAddJarFromHDFS" name="testAddJarFromHDFS"
time="6.73">
    <error type="java.lang.NullPointerException">java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.session.SessionState$JarResourceHook.preHook(SessionState.java:391)
	at org.apache.hadoop.hive.ql.session.SessionState.add_resource(SessionState.java:474)
	at org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:52)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:173)
	at org.apache.hadoop.hive.ql.session.TestAddJarFromHDFS.testAddJarFromHDFS(TestAddJarFromHDFS.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at junit.framework.TestCase.runTest(TestCase.java:154)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at junit.framework.TestSuite.runTest(TestSuite.java:208)
	at junit.framework.TestSuite.run(TestSuite.java:203)
	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422)
	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931)
	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785)
</error>
  </testcase>
  <system-out><![CDATA[Starting DataNode 0 with dfs.data.dir: build/test/data/dfs/data/data1,build/test/data/dfs/data/data2
]]></system-out>
  <system-err><![CDATA[Waiting for the Mini HDFS Cluster to start...
]]></system-err>
</testsuite>
{noformat}

{noformat}
String jarURI = fs.getUri().toString() + "/addJarFromHdfs.jar";
 int ret = cliDriver.processCmd("ADD JAR " + jarURI);
{noformat}

This is a clean checkout and build. I will try to trace this down more.


> UDFs can't be loaded via "add jar" when jar is on HDFS
> ------------------------------------------------------
>
>                 Key: HIVE-1157
>                 URL: https://issues.apache.org/jira/browse/HIVE-1157
>             Project: Hadoop Hive
>          Issue Type: Improvement
>          Components: Query Processor
>            Reporter: Philip Zeyliger
>            Priority: Minor
>         Attachments: hive-1157.patch.txt, HIVE-1157.v2.patch.txt
>
>
> As discussed on the mailing list, it would be nice if you could use UDFs that are on
jars on HDFS.  The proposed implementation would be for "add jar" to recognize that the target
file is on HDFS, copy it locally, and load it into the classpath.
> {quote}
> Hi folks,
> I have a quick question about UDF support in Hive.  I'm on the 0.5 branch.  Can you use
a UDF where the jar which contains the function is on HDFS, and not on the local filesystem.
 Specifically, the following does not seem to work:
> # This is Hive 0.5, from svn
> $bin/hive                                              
> Hive history file=/tmp/philip/hive_job_log_philip_201002081541_370227273.txt
> hive> add jar hdfs://localhost/FooTest.jar;                                      
           
> Added hdfs://localhost/FooTest.jar to class path
> hive> create temporary function cube as 'com.cloudera.FooTestUDF';               
    
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask
> Does this work for other people?  I could probably fix it by changing "add jar" to download
remote jars locally, when necessary (to load them into the classpath), or update URLClassLoader
(or whatever is underneath there) to read directly from HDFS, which seems a bit more fragile.
 But I wanted to make sure that my interpretation of what's going on is right before I have
at it.
> Thanks,
> -- Philip
> {quote}
> {quote}
> Yes that's correct. I prefer to download the jars in "add jar".
> Zheng
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message