hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laxman Ch <laxman....@gmail.com>
Subject Re: Concurrency control
Date Thu, 17 Sep 2015 11:12:15 GMT
Yes. I'm already using cgroups. Cgroups helps in controlling the resources
at container level. But my requirement is more about controlling the
concurrent resource usage of an application at whole cluster level.

And yes, we do configure queues properly. But, that won't help.

For example, I have an application with a requirement of 1000 vcores. But,
I wanted to control this application not to go beyond 100 vcores at any
point of time in the cluster/queue. This makes that application to run
longer even when my cluster is free but I will be able meet the guaranteed
SLAs of other applications.

Hope this helps to understand my question.

And thanks Narasimha for quick response.

On 17 September 2015 at 16:17, Naganarasimha Garla <
naganarasimha.gr@gmail.com> wrote:

> Hi Laxman,
> Yes if cgroups are enabled and "
> yarn.scheduler.capacity.resource-calculator" configured to
> DominantResourceCalculator then cpu and memory can be controlled.
> Please Kindly  furhter refer to the official documentation
> http://hadoop.apache.org/docs/r1.2.1/capacity_scheduler.html
> But may be if say more about problem then we can suggest ideal
> configuration, seems like capacity configuration and splitting of the queue
> is not rightly done or you might refer to Fair Scheduler if you want more
> fairness for container allocation for different apps.
> On Thu, Sep 17, 2015 at 4:10 PM, Laxman Ch <laxman.lux@gmail.com> wrote:
>> Hi,
>> In YARN, do we have any way to control the amount of resources (vcores,
>> memory) used by an application SIMULTANEOUSLY.
>> - In my cluster, noticed some large and long running mr-app occupied all
>> the slots of the queue and blocking other apps to get started.
>> - I'm using Capacity schedulers (using hierarchical queues and preemption
>> disabled)
>> - Using Hadoop version 2.6.0
>> - Did some googling around this and gone through configuration docs but
>> I'm not able to find anything that matches my requirement.
>> If needed, I can provide more details on the usecase and problem.
>> --
>> Thanks,
>> Laxman


View raw message