flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jingsong Li <jingsongl...@gmail.com>
Subject Re: The parallelism of sink is always 1 in sqlUpdate
Date Fri, 06 Mar 2020 10:15:13 GMT
Hi faaron,

For sink parallelism.
- What is parallelism of the input of sink? The sink parallelism should be
- Does you sql have order by or limit ?
Flink batch sql not support range partition now, so it will use single
parallelism to run order by.

For the memory of taskmanager.
There is manage memory option to configure.


Jingsong Lee

On Fri, Mar 6, 2020 at 5:38 PM faaron zheng <faaronzheng@gmail.com> wrote:

> Hi all,
> I am trying to use flink sql to run hive task. I use tEnv.sqlUpdate to
> execute my sql which looks like "insert overtwrite ... select ...". But I
> find the parallelism of sink is always 1, it's intolerable for large data.
> Why it happens? Otherwise, Is there any guide to decide the memory of
> taskmanager when I have two huge table to hashjoin, for example, each table
> has several TB data?
> Thanks,
> Faaron

Best, Jingsong Lee

View raw message