flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: feature request: broadcast POJO objects as part of runtime context
Date Mon, 10 Nov 2014 19:17:26 GMT
But should MyType be serializable or can be of any type?
On Nov 10, 2014 8:06 PM, "Stephan Ewen" <sewen@apache.org> wrote:

> You can put generic objects in the closure as well, just as you can put a
> configuration in the closure.
>
> You can also distribute your objects into the cluster and then use them as
> a broadcast variable:
>
> ------------------------
> DataSet<MyType> aux = env.fromElements(new MyType());
>
>
> someOtherData.map(new MyMapper()).withBroadcastSet(aux, "my broadcast
> data");
>
> ------------------------
>
> See
> http://flink.incubator.apache.org/docs/0.7-incubating/programming_guide.html#broadcast-variables
>
>
>
> Stephan
>
>
> On Mon, Nov 10, 2014 at 7:32 PM, Flavio Pompermaier <pompermaier@okkam.it>
> wrote:
>
>> I think Stefano was asking for a different way to pass a generic
>> Configuration object, not just a subclass of it.
>> For example, in our use case, it would be helpful to broadcast around
>> generic objects/pojos.
>> Is that possible?
>> On Nov 10, 2014 6:46 PM, "Stephan Ewen" <sewen@apache.org> wrote:
>>
>>> Hi!
>>>
>>> I hope I understand correctly what you are trying to do: To have a
>>> config file available in the functions, you can simply do wither of the
>>> following:
>>>
>>> -----------------------
>>> Closure
>>> -----------------------
>>>
>>> Configuration conf = ...
>>>
>>> data.map(new RichMapFunction<String, Integer>() {
>>>
>>>   public void open (Conficuration c) {
>>>      // access the conf object here
>>>      conf.getString(...);
>>>   }
>>>
>>>   public Integer map(String value) {
>>>     // whatever
>>>   }
>>> });
>>> -----------------------
>>>
>>>
>>> -----------------------
>>> Config Parameters
>>> -----------------------
>>>
>>> Configuration conf = ...
>>>
>>> data.map(new RichMapFunction<String, Integer>() {
>>>
>>>   public void open (Conficuration c) {
>>>      // access the c - it will will have all elements of the conf - see
>>> withParameters() below
>>>      c.getString(...);
>>>   }
>>>
>>>   public Integer map(String value) {
>>>     // whatever
>>>   }
>>> })
>>> .withParameters(conf);
>>> -----------------------
>>>
>>>
>>> Stephan
>>>
>>>
>>> On Mon, Nov 10, 2014 at 5:36 PM, Stefano Bortoli <s.bortoli@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> trying to run some legacy code as part of Flink Job, I had to replicate
>>>> configurations files across my cluster. Not a big deal with a small
>>>> cluster, but it would be nice to have these configuration objects
>>>> broadcast-able. Namely, it would be nice to reuse the old "read from conf
>>>> file" logic to build objects that then could be serialized and used along
>>>> the processing through the broadcast mechanism.
>>>>
>>>> Do you think it will be possible? with the new Kryo serialization it
>>>> should not be extremely complicated.
>>>>
>>>> saluti,
>>>> Stefano
>>>>
>>>>
>>>>
>>>
>

Mime
View raw message