flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: How large a Flink cluster can have?
Date Wed, 24 Aug 2016 12:26:14 GMT
There is no recommendation. Generally, it depends on the amount of
processing power and memory that is required for the computation.

A JM/TM ratio doesn't make much sense because there is always only one
active JobManager. Potential others are on standby to become active in
case of failures.

On Wed, Aug 24, 2016 at 2:13 PM, Alexis Gendronneau
<a.gendronneau@gmail.com> wrote:
> Hi,
>
> If possible, with this information, that be great to know  how jobmanager
> has to be scaled according to number of nodes ?
> Will 1 Jobmanager be able to manage hundreds of nodes ? Is there
> recommandation in terms of JM/TM ratio ?
> Thanks
>
> 2016-07-14 15:41 GMT+02:00 Robert Metzger <rmetzger@apache.org>:
>>
>> Hi,
>>
>> I think the reason why this information is not written anywhere is because
>> we don't know it either.
>> Alibaba seems to run a fork of Flink on "thousands of nodes" [1].
>>
>> Maybe some of the production users on this mailing list can share some
>> information regarding this.
>>
>>
>> [1]
>> http://www.slideshare.net/HadoopSummit/improvements-to-flink-its-applications-in-alibaba-search
>>
>>
>> On Wed, Jul 13, 2016 at 3:54 PM, Yan Chou Chen <ycchen915@gmail.com>
>> wrote:
>>>
>>> FAQ[1], mailing list[2], and the powered by page[3] doesn't find
>>> related information. Just out of curiosity, what is the current
>>> largest Flink cluster size running in production? For instance, long
>>> time ago yahoo [4] ran 4k hadoop nodes in production.
>>>
>>> Thanks
>>>
>>> [1]. https://flink.apache.org/faq.html
>>> [2].
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=search_page&node=1&query=max+cluster&i=48
>>> [3]. https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink
>>> [4].
>>> https://developer.yahoo.com/blogs/hadoop/scaling-hadoop-4000-nodes-yahoo-410.html
>>
>>
>
>
>
> --
> Alexis Gendronneau
>
> alexis.gendronneau@corp.ovh.com
> a.gendronneau@gmail.com

Mime
View raw message