hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yufei Gu <flyrain...@gmail.com>
Subject Re: [jira] [Created] (YARN-7672) hadoop-sls can not simulate huge scale of YARN
Date Fri, 22 Dec 2017 06:18:34 GMT
+1 for this idea. It will provide better isolation for scheduler simulating
to split NM simulators or/and AM simulators to different daemons, which can
be placed to different hosts.


On Wed, Dec 20, 2017 at 2:18 AM Jasson Chenwei <ynjassionchen@gmail.com>
wrote:

> It is an interesting point. So you mean you will set up like two daemons,
> one for RM(acts for resource allocation) and the other for NMs(each thread
> represents for a NM ?).  I am also working on a large scale cluster
> simulation project now. I am interested in your configuration for your
> simulation(machine capacity, machine load) and your bottleneck (like if the
> performance is bottlenecked by CPU or the lock-contention or the other
> resources)???
>
>
> Wei Chen
>
> On Mon, Dec 18, 2017 at 11:54 PM, zhangshilong (JIRA) <jira@apache.org>
> wrote:
>
> > zhangshilong created YARN-7672:
> > ----------------------------------
> >
> >              Summary: hadoop-sls can not simulate huge scale of YARN
> >                  Key: YARN-7672
> >                  URL: https://issues.apache.org/jira/browse/YARN-7672
> >              Project: Hadoop YARN
> >           Issue Type: Improvement
> >             Reporter: zhangshilong
> >             Assignee: zhangshilong
> >
> >
> > Our YARN cluster scale to nearly 10 thousands nodes.
> > We need to do scheduler pressure test.
> > we start  2000+ threads to simulate NM and AM. So  cpu.load very high to
> > 100+. I thought that will affect  performance evaluation of scheduler.
> > So I thought to separate the scheduler from the simulator.
> > I start a real RM. Then SLS will register nodes to RM,And submit apps to
> > RM using RM RPC.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.4.14#64029)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>
-- 
Best,

Yufei

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message