impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yu feng <olaptes...@gmail.com>
Subject Re: impala and yarn
Date Tue, 09 May 2017 09:45:59 GMT
Thanks for  your response.  I get you point, We always take 3 or 4 impalad
as coordinator(in about 30 impalad cluster), I do not find any isolation about
CPU. does  Impala community have some plans about this ?

2017-05-09 12:33 GMT+08:00 Matthew Jacobs <mj@cloudera.com>:

> Hi Yu,
>
> Yes, you can use the fair-scheduler.xml and llama-site.xml files as
> you described. Admission control is the best mechanism that Impala has
> right now, unfortunately there isn't a way to get hard resource
> isolation between queries.
>
> I don't understand your problem though, are you saying that more
> queries are running than you expect? That can happen when there are
> many coordinators because admission control isn't centralized -- it
> operates as part of the coordinator. We often recommend using fewer
> coordinators to avoid this "over admission".
>
> On Wed, May 3, 2017 at 7:25 PM, yu feng <olaptestyu@gmail.com> wrote:
> > Thanks for  your response. I have used Impala's admission control in our
> > impala cluster, though we have not deployed CM, we set fair-scheduler.xml
> > and llama-site.xml  and I think it can finish the same goal comparing to
> > using CM.
> >
> > I find in admission control, what we can do is limit Max Running Queries
> > and Max Memory Usage for every queries. and in some scenario, we have
> > to overselling the max memory, such as every impalad has 128GB and we set
> > up two pools, one is max running=32 and max memory =4GB, and another is
> max
> > running=16 and max memory = 2GB, when more and more queries running. we
> > have no ideas about whether the cluster have any problems, maybe we can
> do
> > some tests.
> >
> > what is more, Without Llama, we can not use cgroups achieve isolation
> > between queries(impala run all queries in one process using different
> > threads). I always think this kind of isolation is too softly. do you
> have
> > some suggestion?
> >
> > Thanks a lot
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message