flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Flink ML with DataStream
Date Wed, 19 Jul 2017 19:11:30 GMT
Hi,

unfortunately, it is not possible to convert a DataStream into a DataSet.
Flink's DataSet and DataStream APIs are distinct APIs that cannot be used
together.

The FlinkML library is only available for the DataSet API.
There is some ongoing work to add a machine learning library for streaming
use cases as well, but this is still in an early stage and mostly focusing
on model serving on streams, i.e, applying an externally trained model on
streaming data.

Best, Fabian


2017-07-19 19:07 GMT+02:00 Branham, Jeremy [IT] <Jeremy.D.Branham@sprint.com
>:

> Hello –
>
> I’ve been successful working with Flink in Java, but have some trouble
> trying to leverage the ML library, specifically with KNN.
>
> From my understanding, this is easier in Scala [1] so I’ve been converting
> my code.
>
>
>
> One issue I’ve encountered is – How do I get a DataSet[Vector] from a
> DataStream[MyClass]?
>
> I’ve attempted to use windowing, but scala is completely new to me and I
> may need a push in the right direction.
>
>
>
> The below code executes properly, I’m just unsure of the next step.
>
>
>
>
>
> I’ve also seen an example [2] that looks like something I need to
> implement – especially the PartialModelBuilder.
>
> Am I on the right track?
>
> Thoughts?
>
>
>
> Thanks!
>
>
>
>
>
> [1] - https://stackoverflow.com/questions/44039857/is-there-a-
> apache-flink-machine-learning-tutorial-in-java-language/44040819#44040819
>
> [2] - https://github.com/apache/flink/blob/master/flink-
> examples/flink-examples-streaming/src/main/scala/org/
> apache/flink/streaming/scala/examples/ml/IncrementalLearningSkeleton.scala
>
>
>
>
>
>
>
> Jeremy D. Branham
>
> Technology Architect - Sprint
> O: +1 (972) 405-2970 <(972)%20405-2970> | M: +1 (817) 791-1627
> <(817)%20791-1627>
>
> Jeremy.D.Branham@Sprint.com
>
> #gettingbettereveryday
>
>
>
> ------------------------------
>
> This e-mail may contain Sprint proprietary information intended for the
> sole use of the recipient(s). Any use by others is prohibited. If you are
> not the intended recipient, please contact the sender and delete all copies
> of the message.
>

Mime
View raw message