flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyf...@apache.org>
Subject Re: Modifying start-cluster scripts to efficiently spawn multiple TMs
Date Sun, 10 Jul 2016 21:42:36 GMT
Hi,

I think this would be a nice addition especially for Flink clusters running
on big machines where you might want to run multiple task managers just to
split the memory between multiple java processes.

In any case the previous config format should also be supported as the
default.

I am curious what other developers/users think about this.

Cheers,
Gyula

Saliya Ekanayake <esaliya@gmail.com> ezt írta (időpont: 2016. júl. 9., Szo,
9:44):

> Hi,
>
> The current start/stop scripts SSH worker nodes each time they appear in
> the slaves file. When spawning multiple TMs (like 24 per node), this is
> very inefficient.
>
> I've changed the scripts to do one SSH per node and spawn a given N number
> of TMs afterwards. I can make a pull request if this seems usable to
> others. For now, I assume slaves file will indicate the number of TMs per
> slave in "IP N" format.
>
> Thank you,
> Saliya
>
>
> --
> Saliya Ekanayake
> Ph.D. Candidate | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
>
>

Mime
View raw message