hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Fair scheduler.
Date Wed, 17 Oct 2012 17:53:42 GMT
No, you're right - to define the queue names at the cluster level, the
mapred.queue.names is the right config. To specify a queue at the job
level, mapred.job.queue.name is the right config.

On Wed, Oct 17, 2012 at 11:10 PM, Patai Sangbutsarakum
<silvianhadoop@gmail.com> wrote:
> Harsh.. i am testing it again according to your last instruction.
>
>>> 2. Define your required queues:
>>>mapred.job.queues set to "default,foo,bar" for example, for 3 queues:
>>>default, foo and bar.
>
> From http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u4/cluster_setup.html#Configuring+the+Environment+of+the+Hadoop+Daemons
> I couldn't find "mapred.job.queues" from that link so i have been
> using mapred.queue.names which might be the case that it is my fault.
>
> Please suggest
>
> On Wed, Oct 17, 2012 at 8:43 AM, Harsh J <harsh@cloudera.com> wrote:
>> Hey Robin,
>>
>> Thanks for the detailed post.
>>
>> Just looked at your older thread, and you're right, the JT does write
>> into its system dir for users' job info and token files when
>> initializing the Job. The bug you ran into and the exception+trace you
>> got makes sense now.
>>
>> I just didn't see it on version which Patai seems to be using. I think
>> if he specifies a proper staging directory, he'll go through, cause
>> his trace is different than that of MAPREDUCE-4398 (i.e. system dir
>> vs. staging dir - you had system dir unfortunately).
>>
>> On Wed, Oct 17, 2012 at 8:39 PM, Goldstone, Robin J.
>> <goldstone1@llnl.gov> wrote:
>>> Yes, you would think that users shouldn't need to write to
>>> mapred.system.dir, yet that seems to be the case.  I posted details about
>>> my configuration along with full stack traces last week.  I won't re-post
>>> everything but essentially I have mapred.system.dir defined as a directory
>>> in HDFS owned by mapred:hadoop.  I initially set the permissions to 755
>>> but when the job tracker started up it changed the permissions to 700.
>>> Then when I ran a job as a regular user I got this error:
>>>
>>> 12/10/09 16:27:03 INFO mapred.JobClient: Job Failed: Job initialization
>>> failed:
>>> org.apache.hadoop.security.AccessControlException:
>>> org.apache.hadoop.security.AccessControlException: Permission denied:
>>> user=robing, access=EXECUTE, inode="mapred":mapred:hadoop:rwx------
>>>
>>>
>>> I then manually changed the permissions back to 755 and ran again and got
>>> this error:
>>> 12/10/09 16:31:30 INFO mapred.JobClient: Job Failed: Job initialization
>>> failed:
>>> org.apache.hadoop.security.AccessControlException:
>>> org.apache.hadoop.security.AccessControlException: Permission denied:
>>> user=robing, access=WRITE, inode="mapred":mapred:hadoop:rwxr-xr-x
>>>
>>> I then changed the permissions to 777 and the job ran successfully.  This
>>> suggests that some process was trying to write to write to
>>> mapred.system.dir but did not have sufficient permissions.  The
>>> speculation is that this was being attempted under my uid instead of
>>> mapred.  Perhaps it is something else. I welcome your suggestions.
>>>
>>>
>>> For completeness, I also have mapred.jobtracker.staging.root.dir set to
>>> /user within HDFS.  I can verify the staging files are going there but
>>> something else is still trying to access mapred.system.dir.
>>>
>>> Robin Goldstone, LLNL
>>>
>>> On 10/17/12 12:00 AM, "Harsh J" <harsh@cloudera.com> wrote:
>>>
>>>>Hi,
>>>>
>>>>Regular users never write into the mapred.system.dir AFAICT. That
>>>>directory, is just for the JT to use to mark its presence and to
>>>>"expose" the distributed filesystem it will be relying on.
>>>>
>>>>Users write to their respective staging directories, which lies
>>>>elsewhere and is per-user.
>>>>
>>>>Let me post my environment:
>>>>
>>>>- mapred.system.dir (A HDFS Dir for a JT to register itself) set to
>>>>"/tmp/mapred/system". The /tmp/mapred and /tmp/mapred/system (or
>>>>whatever you configure it to) is to be owned by mapred:hadoop so that
>>>>the JT can feel free to reconfigure it.
>>>>
>>>>- mapreduce.jobtracker.staging.root.dir (A HDFS dir that represents
>>>>the parent directory for user's to write their per-user job stage
>>>>files (JARs, etc.)) is set to "/user". The /user further contains each
>>>>user's home directories, owned all by them. For example:
>>>>
>>>>drwxr-xr-x   - harsh    harsh 0 2012-09-27 15:51 /user/harsh
>>>>
>>>>All staging files from local user 'harsh' are hence written as the
>>>>proper user under /user/harsh/.staging since that user does have
>>>>permissions to write there. For any user to access HDFS, they'd need a
>>>>home directory created on the HDFS by the admin first - and after that
>>>>things users do under their own directory, will work just fine. The JT
>>>>would not have to try to create per-user directories.
>>>>
>>>>On Wed, Oct 17, 2012 at 5:22 AM, Patai Sangbutsarakum
>>>><silvianhadoop@gmail.com> wrote:
>>>>> Thanks everyone, Seem like i hit the dead end.
>>>>> It's kind of funny when i read that jira; run it 4 time and everything
>>>>> will work.. where that magic number from..lol
>>>>>
>>>>> respects
>>>>>
>>>>> On Tue, Oct 16, 2012 at 4:12 PM, Arpit Gupta <arpit@hortonworks.com>
>>>>>wrote:
>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-4398
>>>>>>
>>>>>> is the bug that Robin is referring to.
>>>>>>
>>>>>> --
>>>>>> Arpit Gupta
>>>>>> Hortonworks Inc.
>>>>>> http://hortonworks.com/
>>>>>>
>>>>>> On Oct 16, 2012, at 3:51 PM, "Goldstone, Robin J."
>>>>>><goldstone1@llnl.gov>
>>>>>> wrote:
>>>>>>
>>>>>> This is similar to issues I ran into with permissions/ownership of
>>>>>> mapred.system.dir when using the fair scheduler.  We are instructed
to
>>>>>>set
>>>>>> the ownership of mapred.system.dir to mapred:hadoop and then when
the
>>>>>>job
>>>>>> tracker starts up (running as user mapred) it explicitly sets the
>>>>>> permissions on this directory to 700.  Meanwhile when I go to run
a
>>>>>>job as
>>>>>> a regular user, it is trying to write stuff into mapred.system.dir
but
>>>>>>it
>>>>>> can't due to the ownership/permissions that have been established.
>>>>>>
>>>>>> Per discussion with Arpit Gupta, this is a bug with the fair scheduler
>>>>>>and
>>>>>> it appears from your experience that there are similar issues with
>>>>>> hadoop.tmp.dir.  The whole idea of the fair scheduler is to run jobs
>>>>>>under
>>>>>> the user's identity rather than as user mapred.  This is good from
a
>>>>>> security perspective yet it seems no one bothered to account for
this
>>>>>>in
>>>>>> terms of the permissions that need to be set in the various
>>>>>>directories to
>>>>>> enable this.
>>>>>>
>>>>>> Until this is sorted out by the Hadoop developers, I've put my
>>>>>>attempts to
>>>>>> use the fair scheduler on holdŠ
>>>>>>
>>>>>> Regards,
>>>>>> Robin Goldstone, LLNL
>>>>>>
>>>>>> On 10/16/12 3:32 PM, "Patai Sangbutsarakum" <silvianhadoop@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Harsh,
>>>>>> Thanks for breaking it down clearly. I would say i am successful
98%
>>>>>> from the instruction.
>>>>>> The 2% is about hadoop.tmp.dir
>>>>>>
>>>>>> let's say i have 2 users
>>>>>> userA is a user that start hdfs and mapred
>>>>>> userB is a regular user
>>>>>>
>>>>>> if i use default value of  hadoop.tmp.dir
>>>>>> /tmp/hadoop-${user.name}
>>>>>> I can submit job as usersA but not by usersB
>>>>>> ser=userB, access=WRITE, inode="/tmp/hadoop-userA/mapred/staging"
>>>>>> :userA:supergroup:drwxr-xr-x
>>>>>>
>>>>>> i googled around; someone recommended to change hadoop.tmp.dir to
>>>>>> /tmp/hadoop.
>>>>>> This way it is almost a yay way; the thing is
>>>>>>
>>>>>> if I submit as userA it will create /tmp/hadoop in local machine
which
>>>>>> ownership will be userA.userA,
>>>>>> and once I tried to submit job from the same machine as userB I will
>>>>>> get  "Error creating temp dir in hadoop.tmp.dir /tmp/hadoop due to
>>>>>> Permission denied"
>>>>>> (as because /tmp/hadoop is own by userA.userA). vise versa if I delete
>>>>>> /tmp/hadoop and let the directory be created by userB, userA will
not
>>>>>> be able to submit job.
>>>>>>
>>>>>> Which is the right approach i should work with?
>>>>>> Please suggest
>>>>>>
>>>>>> Patai
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 15, 2012 at 3:18 PM, Harsh J <harsh@cloudera.com>
wrote:
>>>>>>
>>>>>> Hi Patai,
>>>>>>
>>>>>> Reply inline.
>>>>>>
>>>>>> On Tue, Oct 16, 2012 at 2:57 AM, Patai Sangbutsarakum
>>>>>> <silvianhadoop@gmail.com> wrote:
>>>>>>
>>>>>> Thanks for input,
>>>>>>
>>>>>> I am reading the document; i forget to mention that i am on cdh3u4.
>>>>>>
>>>>>>
>>>>>> That version should have the support for all of this.
>>>>>>
>>>>>> If you point your poolname property to mapred.job.queue.name, then
you
>>>>>> can leverage the Per-Queue ACLs
>>>>>>
>>>>>>
>>>>>> Is that mean if i plan to 3 pools of fair scheduler, i have to
>>>>>> configure 3 queues of capacity scheduler. in order to have each pool
>>>>>> can leverage Per-Queue ACL of each queue.?
>>>>>>
>>>>>>
>>>>>> Queues are not hard-tied into CapacityScheduler. You can have generic
>>>>>> queues in MR. And FairScheduler can bind its Pool concept into the
>>>>>> Queue configuration.
>>>>>>
>>>>>> All you need to do is the following:
>>>>>>
>>>>>> 1. Map FairScheduler pool name to reuse queue names itself:
>>>>>>
>>>>>> mapred.fairscheduler.poolnameproperty set to 'mapred.job.queue.name'
>>>>>>
>>>>>> 2. Define your required queues:
>>>>>>
>>>>>> mapred.job.queues set to "default,foo,bar" for example, for 3 queues:
>>>>>> default, foo and bar.
>>>>>>
>>>>>> 3. Define Submit ACLs for each Queue:
>>>>>>
>>>>>> mapred.queue.default.acl-submit-job set to "patai,foobar users,adm"
>>>>>> (usernames groupnames)
>>>>>>
>>>>>> mapred.queue.foo.acl-submit-job set to "spam eggs"
>>>>>>
>>>>>> Likewise for remaining queues, as you need itŠ
>>>>>>
>>>>>> 4. Enable ACLs and restart JT.
>>>>>>
>>>>>> mapred.acls.enabled set to "true"
>>>>>>
>>>>>> 5. Users then use the right API to set queue names before submitting
>>>>>> jobs, or use -Dmapred.job.queue.name=value via CLI (if using Tool):
>>>>>>
>>>>>>
>>>>>>http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/JobCon
>>>>>>f
>>>>>> .html#setQueueName(java.lang.String)
>>>>>>
>>>>>> 6. Done.
>>>>>>
>>>>>> Let us know if this works!
>>>>>>
>>>>>> --
>>>>>> Harsh J
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>>--
>>>>Harsh J
>>>
>>
>>
>>
>> --
>> Harsh J



-- 
Harsh J

Mime
View raw message