flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chan fentes <chanfen...@gmail.com>
Subject OutOfMemoryException: unable to create native thread
Date Tue, 30 Jun 2015 15:17:35 GMT

how many data sources can I use in one Flink plan? Is there any limit? I
get an
java.lang.OutOfMemoryException: unable to create native thread
when having approx. 1500 files. What I basically do is the following:
DataSource ->Map -> Map -> GroupBy -> GroupReduce per file
and then
Union -> GroupBy -> Sum in a tree-like reduction.

I have checked the workflow. It runs on a cluster without any problem, if I
only use few files. Does Flink use a thread per operator? It seems as if I
am limited in the amount of threads I can use. How can I avoid the
exception mentioned above?

Best regards

View raw message