flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: OutOfMemoryException: unable to create native thread
Date Tue, 30 Jun 2015 15:36:11 GMT
Hi Chan,
Flink sources support giving a directory as an input path in a source. If
you do this it will read each of the files in that directory. They way you
do it leads to a very big plan, because the plan will be replicated 1500
times, this could lead to the OutOfMemoryException.

Is there a specific reason why you create 1500 separate sources?

Regards,
Aljoscha

On Tue, 30 Jun 2015 at 17:17 chan fentes <chanfentes@gmail.com> wrote:

> Hello,
>
> how many data sources can I use in one Flink plan? Is there any limit? I
> get an
> java.lang.OutOfMemoryException: unable to create native thread
> when having approx. 1500 files. What I basically do is the following:
> DataSource ->Map -> Map -> GroupBy -> GroupReduce per file
> and then
> Union -> GroupBy -> Sum in a tree-like reduction.
>
> I have checked the workflow. It runs on a cluster without any problem, if
> I only use few files. Does Flink use a thread per operator? It seems as if
> I am limited in the amount of threads I can use. How can I avoid the
> exception mentioned above?
>
> Best regards
> Chan
>

Mime
View raw message