hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Technical question on Capacity Scheduler.
Date Wed, 06 Mar 2013 22:17:59 GMT
That is true *only* if you do not have enough slots to run tasks for other jobs of the same
user in the entire cluster.

On Mar 5, 2013, at 9:02 PM, Jagmohan Chauhan wrote:

> I think i need to re-frame the question a bit. Yes CS supports multiple
> jobs in parallel but of different users. If i am submitting 3 different
> jobs for a same user at the same time, they  are always executed in a FIFO
> manner.
> Am i correct?
> 
> On Tue, Mar 5, 2013 at 8:55 PM, Harsh J <harsh@cloudera.com> wrote:
> 
>> The CS does support running jobs in parallel. Are you observing just
>> the UI or are also noticing a FIFO behavior in logs where assignments
>> can be seen with timestamps?
>> 
>> On Wed, Mar 6, 2013 at 9:03 AM, Jagmohan Chauhan
>> <simplefunduare .mnnit@gmail.com <simplefundumnnit@gmail.com>> wrote:
>>> Hi All
>>> 
>>> Can someone please reply to my queries?
>>> 
>>> On Sun, Mar 3, 2013 at 5:47 PM, Jagmohan Chauhan <
>> simplefundumnnit@gmail.com
>>>> wrote:
>>> 
>>>> Thanks Harsh.
>>>> 
>>>> I have a few more questions.
>>>> 
>>>> Q1: I found it in my experiments using CS that for any user , its next
>> job
>>>> does not start until its current one is finished. Is it true and are
>> there
>>>> any exceptions and if true then why is it so?  I I did not find any such
>>>> condition in the implementation of CS.
>>>> 
>>>> Q2: The concept of reserved slots  is true only if speculative execution
>>>> is on. Am i correct ? If yes,then the code dealing with reserved slots
>> wont
>>>> be executed if speculative execution is off?
>>>> 
>>>> PS: I am working on MRv1.
>>>> 
>>>> 
>>>> On Sun, Mar 3, 2013 at 2:41 AM, Harsh J <harsh@cloudera.com> wrote:
>>>> 
>>>>> On Sun, Mar 3, 2013 at 1:41 PM, Jagmohan Chauhan <
>>>>> simplefundumnnit@gmail.com
>>>>>> wrote:
>>>>> 
>>>>>> Hi
>>>>>> 
>>>>>> I am going through the Capacity Scheduler implementation. There is
>> one
>>>>>> thing i did not understand clearly.
>>>>>> 
>>>>> 
>>>>> Are you reading the YARN CapacityScheduler or the older, MRv1 one? I'd
>>>>> suggest reading the newer one for any implementation or research goals,
>>>>> for
>>>>> it to be more current and future-applicable.
>>>>> 
>>>>> 
>>>>>> 1. Does the o ff-switch task refers to a task in which data has to
be
>>>>>> fetched over the network. It means its not node-local ?
>>>>>> 
>>>>> 
>>>>> Off-switch would imply off-rack, i.e. not node local, nor rack-local.
>>>>> 
>>>>> 
>>>>>> 2. Does off-switch task  includes only the tasks for which map input
>>>>> has to
>>>>>> be fetched from a node on a different rack across the switch or it
>> also
>>>>>> includes task where data has to be fetched from another node on same
>>>>> rack
>>>>>> on same switch?
>>>>>> 
>>>>> 
>>>>> A task's input split is generally supposed to define all locations of
>>>>> available inputs. If the CS is unable to schedule to any of those
>>>>> locations, nor their racks, then it schedules an off-rack (see above)
>> task
>>>>> which has to pull the input from a different rack.
>>>>> 
>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Thanks and Regards
>>>>>> Jagmohan Chauhan
>>>>>> MSc student,CS
>>>>>> Univ. of Saskatchewan
>>>>>> IEEE Graduate Student Member
>>>>>> 
>>>>>> http://homepage.usask.ca/~jac735/
>>>>>> 
>>>>> 
>>>>> Feel free to post any further impl. related questions! :)
>>>>> 
>>>>> --
>>>>> Harsh J
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Thanks and Regards
>>>> Jagmohan Chauhan
>>>> MSc student,CS
>>>> Univ. of Saskatchewan
>>>> IEEE Graduate Student Member
>>>> 
>>>> http://homepage.usask.ca/~jac735/
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Thanks and Regards
>>> Jagmohan Chauhan
>>> MSc student,CS
>>> Univ. of Saskatchewan
>>> IEEE Graduate Student Member
>>> 
>>> http://homepage.usask.ca/~jac735/
>> 
>> 
>> 
>> --
>> Harsh J
>> 
> 
> 
> 
> -- 
> Thanks and Regards
> Jagmohan Chauhan
> MSc student,CS
> Univ. of Saskatchewan
> IEEE Graduate Student Member
> 
> http://homepage.usask.ca/~jac735/

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message