hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (Jira)" <j...@apache.org>
Subject [jira] [Commented] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
Date Mon, 26 Oct 2020 15:53:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220780#comment-17220780
] 

Hadoop QA commented on YARN-10458:
----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  2m 20s{color} |  | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  1s{color} |
 | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
 | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  0s{color} |  |
{color:red} The patch doesn't appear to include any new or modified tests. Please justify
why no new tests are needed for this patch. Also please list what manual steps were performed
to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 25s{color}
|  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  7s{color} |
 | {color:green} trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color}
|
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 55s{color} |
 | {color:green} trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 39s{color}
|  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 58s{color} |
 | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 24s{color}
|  | {color:green} branch has no errors when building and testing our client artifacts. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 41s{color} |
 | {color:green} trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 39s{color} |
 | {color:green} trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
{color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  2m  0s{color} |  |
{color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 58s{color} |
 | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 54s{color}
|  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 55s{color} |
 | {color:green} the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color}
|
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 55s{color} |  |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 53s{color} |
 | {color:green} the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 53s{color} |  |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green}  0m  0s{color} | 
| {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 31s{color}
|  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 52s{color} |
 | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m  5s{color}
|  | {color:green} patch has no errors when building and testing our client artifacts. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 36s{color} |
 | {color:green} the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 33s{color} |
 | {color:green} the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 58s{color} |
 | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 57s{color} | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/249/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt]
| {color:red} hadoop-yarn-server-resourcemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 29s{color}
|  | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 58s{color} |  | {color:black}{color}
|
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoCreatedQueuePreemption
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/249/artifact/out/Dockerfile
|
| JIRA Issue | YARN-10458 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13014125/YARN-10458-001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient
findbugs checkstyle |
| uname | Linux 6169e8dae684 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 74634eb002717a9684d00f0e9dc263ab7eb49246 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
|  Test Results | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/249/testReport/ |
| Max. process+thread count | 820 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/249/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.1.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Hive On Tez queries fails upon submission to dynamically created pools
> ----------------------------------------------------------------------
>
>                 Key: YARN-10458
>                 URL: https://issues.apache.org/jira/browse/YARN-10458
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>            Reporter: Anand Srinivasan
>            Assignee: Peter Bacsko
>            Priority: Major
>         Attachments: YARN-10458-001.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see that the
queue creation fails because ACL submit application check couldn't succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues. For static
queues, this worked but failed for dynamic queues. Also tried setting the below property
but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
Application application_1600399068816_0460 user user1 mapping [default] to [queue1] override
false
> 2020-09-18 01:08:40,579 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAppManager:
User 'user1' from application tag does not have access to  queue 'user1'. The placement is
done for user 'hive'
>  
> Checking the code, scheduler#checkAccess() bails out even before checking the ACL permissions
for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue access-type " + acl + " for queue " + queueName);
}
> return false;                    *<-- the method returns false here.*
> }
> return queue.hasAccess(acl, callerUGI);
> }
> {code}
> As this is an auto created queue, CSQueue may be null in this case. May be scheduler#checkAccess()
should have a logic to differentiate when CSQueue is null and if queue mapping is involved
and if so, check if the parent queue exists and is a managed parent and if so, check if the
parent queue has valid ACL's instead of returning false ?
> Thanks



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message