hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lili Ma <lil...@apache.org>
Subject Re: How about handle stages with different strategies
Date Fri, 15 Jul 2016 02:05:19 GMT
Hi Hubert,

Have some questions about your solution. What you mean is that we shall
have two schedulers, one for slices including scan operator , and the other
for slices not including scan operator?  Then for one query, the two
schedulers will co-work?
I guess what you suggest is that we can assign scan tasks according to the
different type of nodes, say, the disk IO ability for different nodes are
not same, right?  And another possible benefit is that we can different
virtual segments for scan slices and not-scan slices, right? I think the
second one can be converted to M*N dispatching support, say, different
slices can have different virtual segments.

Thanks
Lili

2016-07-14 22:13 GMT+08:00 Hubert Zhang <hzhang@pivotal.io>:

> Hi, all
>   In HAWQ, different stages in HAWQ will be treated as the same. no matter
> from the scheduler view or consider the number of processes.
>   But in some other systems like Presto, There are two schedulers, one is
> sourcePartitionedScheduler used to dispatch scan stage, the other is
> FixedCountScheduler, used to dispatch intermediate stages.
>   I think that one is more flexible. Flexible means that we can write a new
> scanScheduler, which dispatches at split level, for some nodes, which are
> faster than others, will scan more splits than others. This strategy may
> reduce the average IO time.
>    Is there any suggestion?
>
> --
> Thanks
>
> Hubert Zhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message