flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gábor Gévay <gga...@gmail.com>
Subject Re: Dataset split/demultiplex
Date Thu, 12 May 2016 14:28:58 GMT

You can split a DataSet into two DataSets with two filters:

val xs: DataSet[A] = ...
val split1: DataSet[A] = xs.filter(f1)
val split2: DataSet[A] = xs.filter(f2)

where f1 and f2 are true for those elements that should go into the
first and second DataSets respectively. So far, the splits will just
contain elements from the input DataSet, but you can of course apply
some map after one of the filters.

Does this help?


2016-05-12 16:03 GMT+02:00 CPC <achalil@gmail.com>:
> Hi folks,
> Is there any way in dataset api to split Dataset[A] to Dataset[A] and
> Dataset[B] ? Use case belongs to a custom filter component that we want to
> implement. We will want to direct input elements whose result is false
> after we apply the predicate. Actually we want to direct input elements
> that throw exception to another output as well(demultiplexer like
> component).
> Thank you in advance...

View raw message