hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaulboo Zhiao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-5960) yarn capacity DominantResourceCalculator can work on mapreduce , but got error on tez
Date Fri, 02 Dec 2016 08:39:59 GMT
Shaulboo Zhiao created YARN-5960:
------------------------------------

             Summary: yarn capacity DominantResourceCalculator can work on mapreduce , but
got error on tez 
                 Key: YARN-5960
                 URL: https://issues.apache.org/jira/browse/YARN-5960
             Project: Hadoop YARN
          Issue Type: Bug
          Components: capacity scheduler, yarn
    Affects Versions: 2.7.1
         Environment: HDP 2.4.2.0-258 , hadoop 2.7.1 , hive 1.2 , tez 0.7.0 , ranger 0.5.0
            Reporter: Shaulboo Zhiao
            Priority: Minor


Yestoday ,I configured  a queue with 30% resources  , 24core CPU & 1000G memory .  Until
then everything is OK´╝îwhile hql  runs  over  tez engine, ranger-hive is on  and  yarn uses
capacity scheduler .  After I configured 'yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator'
, hql executed failed .The message below : 
INFO  : Session is already open
INFO  : Dag name: select local_code , count(1) fr...local_code(Stage-1)
INFO  : Tez session was closed. Reopening...
INFO  : Session re-established.
INFO  : 

INFO  : Status: Running (Executing on YARN cluster with App id application_1480664840037_0003)

INFO  : Map 1: -/-      Reducer 2: 0/14
INFO  : Map 1: 0/36     Reducer 2: 0/14
INFO  : Map 1: 0(+0,-2)/36      Reducer 2: 0/14
INFO  : Map 1: 0(+0,-4)/36      Reducer 2: 0/14
INFO  : Map 1: 0(+0,-5)/36      Reducer 2: 0/14
INFO  : Map 1: 0(+0,-16)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+0,-35)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+0,-38)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+0,-44)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+0,-50)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+1,-56)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+1,-59)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+2,-63)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+2,-70)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+5,-77)/36     Reducer 2: 0/14
INFO  : Map 1: 0(+0,-84)/36     Reducer 2: 0/14
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1480664840037_0003_1_00, diagnostics=[Task
failed, taskId=task_1480664840037_0003_1_00_000029, diagnostics=[TaskAttempt 0 failed, info=[Container
container_e25_1480664840037_0003_01_000034 finished with diagnostics set to [Container failed,
exitCode=-1000. Application application_1480664840037_0003 initialization failed (exitCode=255)
with output: main : command provided 0
main : run as user is nobody
main : requested yarn user is hive
Requested user nobody is not whitelisted and has id 99,which is below the minimum allowed
500



It seems that yarn run as user hive before , because the ranger is on and ask user hive to
execut all hql  . But I set capacity resouece calculator to  'org.apache.hadoop.yarn.util.resource.DominantResourceCalculator'
,  Yarn  run  task with user 'NOBODY'  , that make the hive run failed everytime  . I change
hive execution engine to mr , everything is OK . As you know,  I need tez to improve performance
over mapreduce , so is it the yarn capacity problem.
 
Feedback would be appreciated!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message