flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: when use broadcast variable and run on bigdata display this error please help
Date Thu, 03 Sep 2015 15:36:40 GMT
Chiwan has a good point. Once the data that needs to be available to all
machines is too large for one machine, there is no good solution any more.
The best approach is an external store to which all nodes have access. It
is not going to be terribly fast, though.

If you are in the situation that you need to broadcast so much data, you
may want to rethink your approach to the problem in the first place. Is
there no solution that can work with partitioned data? Even at the cost of
re-partitioning twice or so?

On Thu, Sep 3, 2015 at 10:35 AM, Chiwan Park <chiwanpark@apache.org> wrote:

> Hi hagersaleh,
>
> Sorry for late reply.
>
> I think using an external system could be a solution for large scale data.
> To use an external system, you have to implement rich functions such as
> RichFilterFunction, RichMapFunction, …, etc.
>
>
> Regards,
> Chiwan Park
>
>
> > On Aug 30, 2015, at 1:30 AM, hagersaleh <loveallah1987@yahoo.com> wrote:
> >
> > where are any ways for use broadcast variable with bigdata
> >
> >
> >
> > --
> > View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/when-use-broadcast-variable-and-run-on-bigdata-display-this-error-please-help-tp2455p2566.html
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive at Nabble.com.
>
>
>
>
>

Mime
View raw message