flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: feature request: broadcast POJO objects as part of runtime context
Date Mon, 10 Nov 2014 20:11:55 GMT
Any type Flink supports as a data type.
Am 10.11.2014 20:19 schrieb "Flavio Pompermaier" <pompermaier@okkam.it>:

> But should MyType be serializable or can be of any type?
> On Nov 10, 2014 8:06 PM, "Stephan Ewen" <sewen@apache.org> wrote:
>
>> You can put generic objects in the closure as well, just as you can put a
>> configuration in the closure.
>>
>> You can also distribute your objects into the cluster and then use them
>> as a broadcast variable:
>>
>> ------------------------
>> DataSet<MyType> aux = env.fromElements(new MyType());
>>
>>
>> someOtherData.map(new MyMapper()).withBroadcastSet(aux, "my broadcast
>> data");
>>
>> ------------------------
>>
>> See
>> http://flink.incubator.apache.org/docs/0.7-incubating/programming_guide.html#broadcast-variables
>>
>>
>>
>> Stephan
>>
>>
>> On Mon, Nov 10, 2014 at 7:32 PM, Flavio Pompermaier <pompermaier@okkam.it
>> > wrote:
>>
>>> I think Stefano was asking for a different way to pass a generic
>>> Configuration object, not just a subclass of it.
>>> For example, in our use case, it would be helpful to broadcast around
>>> generic objects/pojos.
>>> Is that possible?
>>> On Nov 10, 2014 6:46 PM, "Stephan Ewen" <sewen@apache.org> wrote:
>>>
>>>> Hi!
>>>>
>>>> I hope I understand correctly what you are trying to do: To have a
>>>> config file available in the functions, you can simply do wither of the
>>>> following:
>>>>
>>>> -----------------------
>>>> Closure
>>>> -----------------------
>>>>
>>>> Configuration conf = ...
>>>>
>>>> data.map(new RichMapFunction<String, Integer>() {
>>>>
>>>>   public void open (Conficuration c) {
>>>>      // access the conf object here
>>>>      conf.getString(...);
>>>>   }
>>>>
>>>>   public Integer map(String value) {
>>>>     // whatever
>>>>   }
>>>> });
>>>> -----------------------
>>>>
>>>>
>>>> -----------------------
>>>> Config Parameters
>>>> -----------------------
>>>>
>>>> Configuration conf = ...
>>>>
>>>> data.map(new RichMapFunction<String, Integer>() {
>>>>
>>>>   public void open (Conficuration c) {
>>>>      // access the c - it will will have all elements of the conf - see
>>>> withParameters() below
>>>>      c.getString(...);
>>>>   }
>>>>
>>>>   public Integer map(String value) {
>>>>     // whatever
>>>>   }
>>>> })
>>>> .withParameters(conf);
>>>> -----------------------
>>>>
>>>>
>>>> Stephan
>>>>
>>>>
>>>> On Mon, Nov 10, 2014 at 5:36 PM, Stefano Bortoli <s.bortoli@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> trying to run some legacy code as part of Flink Job, I had to
>>>>> replicate configurations files across my cluster. Not a big deal with
a
>>>>> small cluster, but it would be nice to have these configuration objects
>>>>> broadcast-able. Namely, it would be nice to reuse the old "read from
conf
>>>>> file" logic to build objects that then could be serialized and used along
>>>>> the processing through the broadcast mechanism.
>>>>>
>>>>> Do you think it will be possible? with the new Kryo serialization it
>>>>> should not be extremely complicated.
>>>>>
>>>>> saluti,
>>>>> Stefano
>>>>>
>>>>>
>>>>>
>>>>
>>

Mime
View raw message