hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11211) mapreduce.job.classloader.system.classes property behaves differently when the exclusion and inclusion order is different
Date Mon, 20 Oct 2014 20:51:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177453#comment-14177453
] 

Sangjin Lee commented on HADOOP-11211:
--------------------------------------

Thanks for reporting this [~timyitong]!

Actually this needs a little discussion. I agree that the current behavior is not as well
documented as it should. As you stated, it is a "first-match-wins" behavior.

What you propose also seems reasonable, but needs more clarification. Your proposal is to
disregard the order and consider all matches and do their logical AND's. But there can be
cases where that rule is counter-intuitive. In your example, if you have "foo.bar.,-foo.bar."
it would mean that the class is not considered a system class. Does that mean that an exclusion
match is stronger than an inclusion match?

Here is an even more interesting case: how about "foo.bar.tar.,-foo.bar."? One reasonable
interpretation of this may be that "exclude everything that belongs to foo.bar. but still
include foo.bar.tar.". However, using the AND rule, it would mean that all classes under "foo.bar"
would be excluded and the "foo.bar.tar." rule would be effectively ignored.

Also, how about a exact match (e.g. "foo.bar.MyClass")? Would it need to be considered over
package matches?

> mapreduce.job.classloader.system.classes property behaves differently when the exclusion
and inclusion order is different
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-11211
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11211
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Yitong Zhou
>            Assignee: Yitong Zhou
>
> If we want to include package foo.bar.* but exclude all sub packages named foo.bar.tar.*
in system classes, configuring "mapreduce.job.classloader.system.classes=foo.bar.,-foo.bar.tar."
won't work. foo.bar.tar will still be pulled in. But if we change the order:
> "mapreduce.job.classloader.system.classes=-foo.bar.tar.,foo.bar.", then it will work.
> This bug is due to the implementation of ApplicationClassLoaser#isSystemClass in hadoop-common,
where we simply return the matching result immediately when the class name hits the first
match (either positive or negative).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message