flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chiwan Park <chiwanp...@apache.org>
Subject Re: when use broadcast variable and run on bigdata display this error please help
Date Sat, 05 Sep 2015 13:13:36 GMT
Hi hagersaleh,

You should know why the error occurred with large scale data. Broadcast variables can handle
only data of which size is fit for single machine.

I meant that using an external system such as Redis, HBase, …, etc. The connection with
the external system could be initialized in `open` method of rich functions such as `RichFilterFunction`,
`RichFlatMapFunction`, …, etc..

You can choose another solution which Stephen said. He said that rethink your approach. I
think that rethinking your algorithm would be better than my suggestion.

From your code, I don’t understand why you want to use broadcast variable. You can do same
thing with filter and join operations. Here is my implementation [1].

Chiwan Park

[1] https://gist.github.com/chiwanpark/a0b0269c9a9b058d15d3

> On Sep 4, 2015, at 3:51 AM, hagersaleh <loveallah1987@yahoo.com> wrote:
> Hi Chiwan Park 
> not understand this solution please explain more
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/when-use-broadcast-variable-and-run-on-bigdata-display-this-error-please-help-tp2455p2676.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

View raw message