ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zack Marsh (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMBARI-12402) Permission denied errors for local usercache directories when attempting to run MapReduce job on Kerberos enabled cluster
Date Mon, 13 Jul 2015 22:19:05 GMT
Zack Marsh created AMBARI-12402:
-----------------------------------

             Summary: Permission denied errors for local usercache directories when attempting
to run MapReduce job on Kerberos enabled cluster
                 Key: AMBARI-12402
                 URL: https://issues.apache.org/jira/browse/AMBARI-12402
             Project: Ambari
          Issue Type: Bug
    Affects Versions: 2.1.0
         Environment: sles11sp3
            Reporter: Zack Marsh


Prior to enabling Kerberos on the Hadoop cluster, I am able to run a simple MapReduce example
as the Linux user 'tdatuser':
{code}
iripiri1:~ # su tdatuser
    tdatuser@piripiri1:/root> yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-2.*.jar
pi 16 10000
    Number of Maps  = 16
    Samples per Map = 10000
    Wrote input for Map #0
    Wrote input for Map #1
    Wrote input for Map #2
    Wrote input for Map #3
    Wrote input for Map #4
    Wrote input for Map #5
    Wrote input for Map #6
    Wrote input for Map #7
    Wrote input for Map #8
    Wrote input for Map #9
    Wrote input for Map #10
    Wrote input for Map #11
    Wrote input for Map #12
    Wrote input for Map #13
    Wrote input for Map #14
    Wrote input for Map #15
    Starting Job
    15/07/13 17:02:31 INFO impl.TimelineClientImpl: Timeline service address: http:/     
                             s/v1/timeline/
    15/07/13 17:02:31 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to
    15/07/13 17:02:31 INFO input.FileInputFormat: Total input paths to process : 16
    15/07/13 17:02:31 INFO mapreduce.JobSubmitter: number of splits:16
    15/07/13 17:02:31 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_14
    15/07/13 17:02:32 INFO impl.YarnClientImpl: Submitted application application_14
    15/07/13 17:02:32 INFO mapreduce.Job: The url to track the job: http://piripiri3     
                             cation_1436821014431_0003/
    15/07/13 17:02:32 INFO mapreduce.Job: Running job: job_1436821014431_0003
    15/07/13 17:05:50 INFO mapreduce.Job: Job job_1436821014431_0003 running in uber mode
: false
    15/07/13 17:05:50 INFO mapreduce.Job:  map 0% reduce 0%
    15/07/13 17:05:56 INFO mapreduce.Job:  map 6% reduce 0%
    15/07/13 17:06:00 INFO mapreduce.Job:  map 13% reduce 0%
    15/07/13 17:06:01 INFO mapreduce.Job:  map 38% reduce 0%
    15/07/13 17:06:05 INFO mapreduce.Job:  map 44% reduce 0%
    15/07/13 17:06:07 INFO mapreduce.Job:  map 63% reduce 0%
    15/07/13 17:06:09 INFO mapreduce.Job:  map 69% reduce 0%
    15/07/13 17:06:11 INFO mapreduce.Job:  map 75% reduce 0%
    15/07/13 17:06:12 INFO mapreduce.Job:  map 81% reduce 0%
    15/07/13 17:06:13 INFO mapreduce.Job:  map 81% reduce 25%
    15/07/13 17:06:14 INFO mapreduce.Job:  map 94% reduce 25%
    15/07/13 17:06:16 INFO mapreduce.Job:  map 100% reduce 31%
    15/07/13 17:06:17 INFO mapreduce.Job:  map 100% reduce 100%
    15/07/13 17:06:17 INFO mapreduce.Job: Job job_1436821014431_0003 completed successfully
    15/07/13 17:06:17 INFO mapreduce.Job: Counters: 49
            File System Counters
                    FILE: Number of bytes read=358
                    FILE: Number of bytes written=2249017
                    FILE: Number of read operations=0
                    FILE: Number of large read operations=0
                    FILE: Number of write operations=0
                    HDFS: Number of bytes read=4198
                    HDFS: Number of bytes written=215
                    HDFS: Number of read operations=67
                    HDFS: Number of large read operations=0
                    HDFS: Number of write operations=3
            Job Counters
                    Launched map tasks=16
                    Launched reduce tasks=1
                    Data-local map tasks=16
                    Total time spent by all maps in occupied slots (ms)=160498
                    Total time spent by all reduces in occupied slots (ms)=27302
                    Total time spent by all map tasks (ms)=80249
                    Total time spent by all reduce tasks (ms)=13651
                    Total vcore-seconds taken by all map tasks=80249
                    Total vcore-seconds taken by all reduce tasks=13651
                    Total megabyte-seconds taken by all map tasks=246524928
                    Total megabyte-seconds taken by all reduce tasks=41935872
            Map-Reduce Framework
                    Map input records=16
                    Map output records=32
                    Map output bytes=288
                    Map output materialized bytes=448
                    Input split bytes=2310
                    Combine input records=0
                    Combine output records=0
                    Reduce input groups=2
                    Reduce shuffle bytes=448
                    Reduce input records=32
                    Reduce output records=0
                    Spilled Records=64
                    Shuffled Maps =16
                    Failed Shuffles=0
                    Merged Map outputs=16
                    GC time elapsed (ms)=1501
                    CPU time spent (ms)=13670
                    Physical memory (bytes) snapshot=13480296448
                    Virtual memory (bytes) snapshot=72598511616
                    Total committed heap usage (bytes)=12508463104
            Shuffle Errors
                    BAD_ID=0
                    CONNECTION=0
                    IO_ERROR=0
                    WRONG_LENGTH=0
                    WRONG_MAP=0
                    WRONG_REDUCE=0
            File Input Format Counters
                    Bytes Read=1888
            File Output Format Counters
                    Bytes Written=97
    Job Finished in 226.813 seconds
    Estimated value of Pi is 3.14127500000000000000
{code}

However, after enabling Kerberos, the job fails:
{code}
tdatuser@piripiri1:/root> kinit -kt /etc/security/keytabs/tdatuser.headless.keytab tdatuser
    tdatuser@piripiri1:/root> yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-2.*.jar
pi 16 10000
    Number of Maps  = 16
    Samples per Map = 10000
    Wrote input for Map #0
    Wrote input for Map #1
    Wrote input for Map #2
    Wrote input for Map #3
    Wrote input for Map #4
    Wrote input for Map #5
    Wrote input for Map #6
    Wrote input for Map #7
    Wrote input for Map #8
    Wrote input for Map #9
    Wrote input for Map #10
    Wrote input for Map #11
    Wrote input for Map #12
    Wrote input for Map #13
    Wrote input for Map #14
    Wrote input for Map #15
    Starting Job
    15/07/13 17:27:05 INFO impl.TimelineClientImpl: Timeline service address: http://piripiri1.labs.teradata.com:8188/ws/v1/timeline/
    15/07/13 17:27:05 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 140 for tdatuser
on ha-hdfs:PIRIPIRI
    15/07/13 17:27:05 INFO security.TokenCache: Got dt for hdfs://PIRIPIRI; Kind: HDFS_DELEGATION_TOKEN,
Service: ha-hdfs:PIRIPIRI, Ident: (HDFS_DELEGATION_TOKEN token 140 for tdatuser)
    15/07/13 17:27:06 INFO input.FileInputFormat: Total input paths to process : 16
    15/07/13 17:27:06 INFO mapreduce.JobSubmitter: number of splits:16
    15/07/13 17:27:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1436822321287_0007
    15/07/13 17:27:06 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:PIRIPIRI,
Ident: (HDFS_DELEGATION_TOKEN token 140 for tdatuser)
    15/07/13 17:27:06 INFO impl.YarnClientImpl: Submitted application application_1436822321287_0007
    15/07/13 17:27:06 INFO mapreduce.Job: The url to track the job: http://piripiri2.labs.teradata.com:8088/proxy/application_1436822321287_0007/
    15/07/13 17:27:06 INFO mapreduce.Job: Running job: job_1436822321287_0007
    15/07/13 17:27:09 INFO mapreduce.Job: Job job_1436822321287_0007 running in uber mode
: false
    15/07/13 17:27:09 INFO mapreduce.Job:  map 0% reduce 0%
    15/07/13 17:27:09 INFO mapreduce.Job: Job job_1436822321287_0007 failed with state FAILED
due to: Application application_1436822321287_0007 failed 2 times due to AM Container for
appattempt_1436822321287_0007_000002 exited with  exitCode: -1000
    For more detailed output, check application tracking page:http://piripiri2.labs.teradata.com:8088/cluster/app/application_1436822321287_0007Then,
click on links to logs of each attempt.
    Diagnostics: Application application_1436822321287_0007 initialization failed (exitCode=255)
with output: main : command provided 0
    main : run as user is tdatuser
    main : requested yarn user is tdatuser
    Can't create directory /data1/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data2/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data3/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data4/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data5/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data6/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data7/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data8/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data9/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data10/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data11/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Can't create directory /data12/hadoop/yarn/local/usercache/tdatuser/appcache/application_1436822321287_0007
- Permission denied
    Did not create any app directories

    Failing this attempt. Failing the application.
    15/07/13 17:27:09 INFO mapreduce.Job: Counters: 0
    Job Finished in 4.748 seconds
    java.io.FileNotFoundException: File does not exist: hdfs://PIRIPIRI/user/tdatuser/QuasiMonteCarlo_1436822823095_2120947622/out/reduce-out
            at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
            at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
            at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
            at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
            at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1752)
            at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1776)
            at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
            at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
            at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:483)
            at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
            at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
            at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:483)
            at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{code}

As seen above there are many "Can't create directory... Permission denied errors" related
to the local usercache directory for the 'tdatuser'.

Prior to enabling Kerberos, the contents of a usercache directory was as follows:
{code}
piripiri4:~ # ls -l /data1/hadoop/yarn/local/usercache/
    total 0
    drwxr-xr-x 3 yarn hadoop 21 Jul 13 16:59 ambari-qa
    drwxr-x--- 4 yarn hadoop 37 Jul 13 17:00 tdatuser
{code}

After enabling Kerberos the contents are:
{code}
piripiri4:~ # ls -l /data1/hadoop/yarn/local/usercache/
    total 0
    drwxr-s--- 4 ambari-qa hadoop 37 Jul 13 17:21 ambari-qa
    drwxr-x--- 4 yarn      hadoop 37 Jul 13 17:00 tdatuser
{code}

It appears that the owner of the usercache directory for the 'ambari-qa' user was updated,
but the 'tdatuser' directory was not. 

Is this expected behavior, and is there a recommended work-around for this issue? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message