hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (JIRA)" <>
Subject [jira] [Commented] (HIVE-3607) Set mapreduce.task.classpath.user.precedence to true by default
Date Mon, 29 Oct 2012 06:31:15 GMT


Namit Jain commented on HIVE-3607:

This seems to be logical - hive jars should always come before hadoop server side jars. What
is the reason for giving preference to hadoop server jars ?
Is it the issue that the client can submit malicious hadoop jars - anyway, the client today
can submit any jar and write a UDF which accesses those jars.

If we want to be conservative, we can come up with a blacklist of hadoop jars which cannot
be provided by hive.
> Set mapreduce.task.classpath.user.precedence to true by default
> ---------------------------------------------------------------
>                 Key: HIVE-3607
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: Configuration
>    Affects Versions: 0.10.0
>            Reporter: Kevin Wilfong
>            Assignee: Kevin Wilfong
> When queries are actually run in a Hadoop cluster, Hive's jars are appended to Hadoop's
classpath.  However, when we test/run jobs locally Hive's classpath comes first.  This leads
to issues like the one brought up here after the patch was committed HIVE-3581 where a change
depended on a jar Hive includes which conflicted with one provided by Hadoop which is an older
version in 0.20
> It's possible that more of the jars we include are getting preceded by older jars in
Hadoop, and we haven't noticed yet.
> If we add Hive jars to the beginning of Hadoop's classpath we will be in control in such
situations where the jars are backwards compatible.  We will be able to update the jars in
Hive and these will be used at run time, instead of just compile time.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message