flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: DeltaIterations: shrink solution set
Date Wed, 11 Feb 2015 09:01:49 GMT
You can also use a bulk iteration and just keep the state yourself. Since
the functions love across iterations, it is easily doable to just gather
the state in a HashMap yourself. Use map(), or mapPartition(), a manual
partition() call - that should do the trick...
Am 10.02.2015 21:44 schrieb "Alexander Alexandrov" <
alexander.s.alexandrov@gmail.com>:

> True.
>
> 2015-02-10 19:14 GMT+01:00 Vasiliki Kalavri <vasilikikalavri@gmail.com>:
>
>> Hi,
>>
>> It's hard to tell without details about your algorithm, but what you're
>> describing sounds to me like something you can use the workset for.
>>
>> -V.
>> On Feb 10, 2015 6:54 PM, "Alexander Alexandrov" <
>> alexander.s.alexandrov@gmail.com> wrote:
>>
>>> I am not sure whether this is supported at the moment. The only
>>> workaround I could think of is indeed to use a boolean flag that indicates
>>> whether the element has been deleted or not.
>>>
>>> An alternative approach is to ditch Flink's native iteration construct
>>> and write your intermediate results to Tachyon or HDFS after each iteration
>>> using the TypeInfoInput/OutputFormats. You then have full control how the
>>> old and the new solutions sets should be merged.
>>>
>>> BTW can you share some details about that particular algorithm? I was
>>> thinking about examples iterative algorithms with this property...
>>>
>>> Regards,
>>> A.
>>>
>>>
>>> 2015-02-10 14:18 GMT+01:00 Kruse, Sebastian <Sebastian.Kruse@hpi.de>:
>>>
>>>>  Hi everyone,
>>>>
>>>>
>>>>
>>>> From playing around a bit around with delta iterations, I saw that you
>>>> can update elements from the solution set and add new elements. My question
>>>> is: is it possible to remove elements from the solution set (apart from
>>>> marking them as “deleted” somehow)?
>>>>
>>>>
>>>>
>>>> My use case at hand for this is the following: In each iteration, I
>>>> generate candidate solutions that I want to verify within the next
>>>> iteration. If verification fails, I would like to remove them from the
>>>> solution set, otherwise retain them.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Sebastian
>>>>
>>>
>>>
>

Mime
View raw message