hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4336) YARN NodeManager - Container Initialization - Excessive load on NSS/LDAP
Date Fri, 06 Nov 2015 15:46:27 GMT

    [ https://issues.apache.org/jira/browse/YARN-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993836#comment-14993836
] 

Jason Lowe commented on YARN-4336:
----------------------------------

I believe this is a duplicate of YARN-3452.  We fixed it by reverting  HADOOP-10650 in our
internal build since we don't need the blacklisting functionality added by that feature, and
that's what caused the excess lookups.  IMHO the real fix is to have YARN not use bogus user
names, but I don't know if that's going to be an easy change to make.

> YARN NodeManager - Container Initialization - Excessive load on NSS/LDAP
> ------------------------------------------------------------------------
>
>                 Key: YARN-4336
>                 URL: https://issues.apache.org/jira/browse/YARN-4336
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.4.0, 2.4.1, 2.6.0, 2.7.0, 2.6.1, 2.7.1
>         Environment: NSS w/ SSSD or Dell/Quest - VASD
>            Reporter: Greg Senia
>            Assignee: Greg Senia
>         Attachments: YARN-4336-tactical.txt
>
>
> Hi folks after performing some debug for our Unix Engineering and Active Directory teams
it was discovered that on YARN Container Initialization a call via Hadoop Common AccessControlList.java:
>   for(String group: ugi.getGroupNames()) {
>         if (groups.contains(group)) {
>           return true;
>         }
>       }
> Unfortunately with the security call to check access on "appattempt_XXXXXXXXXXXXX_XXXXX_XXXXX"
will always return false but will make unnecessary calls to NameSwitch service on linux which
will call things like SSSD/Quest VASD which will then initiate LDAP calls looking for non
existent userid's causing excessive load on LDAP.
> For now our tactical work around is as follows:
> /**
>    * Checks if a user represented by the provided {@link UserGroupInformation}
>    * is a member of the Access Control List
>    * @param ugi UserGroupInformation to check if contained in the ACL
>    * @return true if ugi is member of the list
>    */
>   public final boolean isUserInList(UserGroupInformation ugi) {
>     if (allAllowed || users.contains(ugi.getShortUserName())) {
>       return true;
>     } else {
>         String patternString = "^appattempt_\\d+_\\d+_\\d+$";
>         Pattern pattern = Pattern.compile(patternString);
>         Matcher matcher = pattern.matcher(ugi.getShortUserName());
>         boolean matches = matcher.matches();
>         if (matches) {
>         	LOG.debug("Bailing !! AppAttempt Matches DONOT call UGI FOR GROUPS!!");;
>         	return false;
>         }
>     	
>     	
>       for(String group: ugi.getGroupNames()) {
>         if (groups.contains(group)) {
>           return true;
>         }
>       }
>     }
>     return false;
>   }
>   public boolean isUserAllowed(UserGroupInformation ugi) {
>     return isUserInList(ugi);
>   }
> Example of VASD Debug log showing the lookups for one task attempt 32 of them:
> One task:
> Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:55:43 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:55:43 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:56:15 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:56:15 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:56:45 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:56:45 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:57:18 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:57:18 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:57:49 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:57:49 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:58:22 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:58:22 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:58:52 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:58:52 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:59:30 xhadoopm5d vasd[20741]: _vasug_user_namesearch_gc: searching GC for host
service domain EXNSD.EXA.EXAMPLE.COM with filter (&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))
> Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>
> Oct 30 22:59:30 xhadoopm5d vasd[20741]: libvas_attrs_find_uri: Searching <GC://@EXNSD.EXA.EXAMPLE.COM>
with filter=<(&(objectCategory=Person)(samaccountname=appattempt_1446145939879_0022_000001))>,
base=<>, scope=<sub>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message