flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: How to register aggregation convergence criterion to bulk iteration in scala API?
Date Tue, 02 Feb 2016 20:51:56 GMT
You are right, that is currently missing in the Scala API. Would be good to
add this, for feature completeness in the Scala API.

As a workaround for now: Can you access the Java IterativeDataSet from the
Scala data set, and register it there?

Greetings,
Stephan


On Thu, Jan 28, 2016 at 11:05 PM, Fridtjof Sander <
fsander@mailbox.tu-berlin.de> wrote:

> Hi,
>
> I want to register a custom aggregation convergence criterion to a bulk
> iteration and I want to use the scala API.
> It appears to me that this is not possible at the moment, right?
>
> The AggregatorRegistry is exposed by IterativeDataSet.java, which is
> hidden by DataSet.scala:
>
>   def iterate(maxIterations: Int)(stepFunction: (DataSet[T]) =>
> DataSet[T]): DataSet[T] = {
>     val iterativeSet =
>       new IterativeDataSet[T](
>         javaSet.getExecutionEnvironment,
>         javaSet.getType,
>         javaSet,
>         maxIterations)
>
>     val resultSet = stepFunction(wrap(iterativeSet))
>     val result = iterativeSet.closeWith(resultSet.javaSet)
>     wrap(result)
>   }
>
> I am aware of the iterateWithTermination-possibility and it's a
> work-around for me, but I guess the aggregated convergence criterion would
> be more efficient.
> Am I missing something?
>
> Best,
> Fridtjof
>

Mime
View raw message