flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: feature request: broadcast POJO objects as part of runtime context
Date Mon, 10 Nov 2014 19:05:02 GMT
You can put generic objects in the closure as well, just as you can put a
configuration in the closure.

You can also distribute your objects into the cluster and then use them as
a broadcast variable:

------------------------
DataSet<MyType> aux = env.fromElements(new MyType());


someOtherData.map(new MyMapper()).withBroadcastSet(aux, "my broadcast
data");

------------------------

See
http://flink.incubator.apache.org/docs/0.7-incubating/programming_guide.html#broadcast-variables



Stephan


On Mon, Nov 10, 2014 at 7:32 PM, Flavio Pompermaier <pompermaier@okkam.it>
wrote:

> I think Stefano was asking for a different way to pass a generic
> Configuration object, not just a subclass of it.
> For example, in our use case, it would be helpful to broadcast around
> generic objects/pojos.
> Is that possible?
> On Nov 10, 2014 6:46 PM, "Stephan Ewen" <sewen@apache.org> wrote:
>
>> Hi!
>>
>> I hope I understand correctly what you are trying to do: To have a config
>> file available in the functions, you can simply do wither of the following:
>>
>> -----------------------
>> Closure
>> -----------------------
>>
>> Configuration conf = ...
>>
>> data.map(new RichMapFunction<String, Integer>() {
>>
>>   public void open (Conficuration c) {
>>      // access the conf object here
>>      conf.getString(...);
>>   }
>>
>>   public Integer map(String value) {
>>     // whatever
>>   }
>> });
>> -----------------------
>>
>>
>> -----------------------
>> Config Parameters
>> -----------------------
>>
>> Configuration conf = ...
>>
>> data.map(new RichMapFunction<String, Integer>() {
>>
>>   public void open (Conficuration c) {
>>      // access the c - it will will have all elements of the conf - see
>> withParameters() below
>>      c.getString(...);
>>   }
>>
>>   public Integer map(String value) {
>>     // whatever
>>   }
>> })
>> .withParameters(conf);
>> -----------------------
>>
>>
>> Stephan
>>
>>
>> On Mon, Nov 10, 2014 at 5:36 PM, Stefano Bortoli <s.bortoli@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> trying to run some legacy code as part of Flink Job, I had to replicate
>>> configurations files across my cluster. Not a big deal with a small
>>> cluster, but it would be nice to have these configuration objects
>>> broadcast-able. Namely, it would be nice to reuse the old "read from conf
>>> file" logic to build objects that then could be serialized and used along
>>> the processing through the broadcast mechanism.
>>>
>>> Do you think it will be possible? with the new Kryo serialization it
>>> should not be extremely complicated.
>>>
>>> saluti,
>>> Stefano
>>>
>>>
>>>
>>

Mime
View raw message