hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HIVE-574) Hive should use ClassLoader from hadoop Configuration
Date Wed, 24 Jun 2009 08:37:07 GMT

     [ https://issues.apache.org/jira/browse/HIVE-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zheng Shao updated HIVE-574:
----------------------------

    Attachment: HIVE-574.2.patch

This patch fixed a bug in ExecDriver.initialize where we were setting "tmpjars", which were
overwritten by ExecDriver.execute().

> there's a bunch of places where JavaUtils.getClassLoader() is called - all of these need
to use the conf classloader instead.
I tried to change all of these to use conf classloader, but in lots of places there are no
shared conf objects, so I will leave it for now. I changed the comment so it's clear that
in hive (outside hadoop map-reduce jobs) we are using thread classloader, while in hadoop
map-reduce jobs we are using conf classloaders.

> where are we invoking conf.setClassLoader? (addToClassPath should be calling this it
seems)
I moved that logic outside of the addToClassPath function. All callers will do that set.

> if conf.set/getClassLoader is supported in all versions 17 onwards - then we can just
remove the -libjars business - since that doesn't help us any bit now (ExecDriver is not a
Tool - so it doesn't even get the conf with the classpath set that a Tool would normally get
by specifying the -libjars option).

Maybe in the future we should move ExecDriver to a tool? I think we should rely more on hadoop
to set the classpath for us, so maybe in the future we should remove the special "if (local)"
block in ExecDriver where we set the classpath ourselves.


> Hive should use ClassLoader from hadoop Configuration
> -----------------------------------------------------
>
>                 Key: HIVE-574
>                 URL: https://issues.apache.org/jira/browse/HIVE-574
>             Project: Hadoop Hive
>          Issue Type: Bug
>    Affects Versions: 0.3.0, 0.3.1
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>         Attachments: HIVE-574.1.patch, HIVE-574.2.patch
>
>
> See HIVE-338.
> Hive should always use the getClassByName method from hadoop Configuration, so that we
choose the correct ClassLoader. Examples include all plug-in interfaces, including UDF/GenericUDF/UDAF,
SerDe, and FileFormats. Basically the following code snippet shows the idea:
> {code}
> package org.apache.hadoop.conf;
> public class Configuration implements Iterable<Map.Entry<String,String>>
{
>    ...
>   /**
>    * Load a class by name.
>    * 
>    * @param name the class name.
>    * @return the class object.
>    * @throws ClassNotFoundException if the class is not found.
>    */
>   public Class<?> getClassByName(String name) throws ClassNotFoundException {
>     return Class.forName(name, true, classLoader);
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message