Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <0E0AC9FC-9100-45CD-BC14-498E4AE4C1EB@gmail.com>
References: <0E0AC9FC-9100-45CD-BC14-498E4AE4C1EB@gmail.com>
From: Fabian Hueske <fhueske@gmail.com>
Date: Tue, 17 Oct 2017 10:54:49 +0200
Message-ID: <CAAdrtT2DOjv=8s-erS+mXyJU1PnaYY6d1+X0DoQf7AXbBd2U_A@mail.gmail.com>
Subject: Re: Split a dataset
To: Magnus Vojbacke <magnus.vojbacke@gmail.com>
Cc: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="f403045e630c0677fe055bba4935"
archived-at: Tue, 17 Oct 2017 08:55:44 -0000

--f403045e630c0677fe055bba4935
Content-Type: text/plain; charset="UTF-8"

Hi Magnus,

there is no Split operator on the DataSet API.

As you said, this can be done using a FilterFunction. This also allows for
non-binary splits:

DataSet<X> setToSplit = ...
DataSet<X> firstSplit = setToSplit.filter(new SplitCondition1());
DataSet<X> secondSplit = setToSplit.filter(new SplitCondition2());
DataSet<X> thirdSplit = setToSplit.filter(new SplitCondition3());

where SplitCondition1, SplitCondition2, and SplitCondition3 are
FilterFunction that filter out all records that don't belong to the split.

Best, Fabian

2017-10-17 10:42 GMT+02:00 Magnus Vojbacke <magnus.vojbacke@gmail.com>:

> I'm looking for something like DataStream.split(), but for DataSets. I'd
> like to split my streaming data so messages go to different parts of an
> execution graph, based on arbitrary logic.
>
> DataStream.split() seems to be perfect, except that my source is a CSV
> file, and I have only found built in functions for reading CSV files into a
> DataSet.
>
> I've evaluated using DataSet.filter(), but as far as I can tell, that only
> allows me to emulate a yes/no split. This is not ideal because it's too
> coarse, and I would prefer a more fine grained split than that.
>
>
> Do you have any suggestions on how I can achieve my arbitrary splitting
> logic for a) DataSets in general, or b) CSV files?
>
>

--f403045e630c0677fe055bba4935
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div>Hi Magnus,<br><br></div>ther=
e is no Split operator on the DataSet API.<br><br></div>As you said, this c=
an be done using a FilterFunction. This also allows for non-binary splits:<=
br><br></div>DataSet&lt;X&gt; setToSplit =3D ...<br></div>DataSet&lt;X&gt; =
firstSplit =3D setToSplit.filter(new SplitCondition1());<br>DataSet&lt;X&gt=
; secondSplit =3D setToSplit.filter(new SplitCondition2());<br>DataSet&lt;X=
&gt; thirdSplit =3D setToSplit.filter(new SplitCondition3());<br><br></div>=
where SplitCondition1, SplitCondition2, and SplitCondition3 are FilterFunct=
ion that filter out all records that don&#39;t belong to the split.<br><br>=
</div>Best, Fabian<br></div><div class=3D"gmail_extra"><br><div class=3D"gm=
ail_quote">2017-10-17 10:42 GMT+02:00 Magnus Vojbacke <span dir=3D"ltr">&lt=
;<a href=3D"mailto:magnus.vojbacke@gmail.com" target=3D"_blank">magnus.vojb=
acke@gmail.com</a>&gt;</span>:<br><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I&#39;m =
looking for something like DataStream.split(), but for DataSets. I&#39;d li=
ke to split my streaming data so messages go to different parts of an execu=
tion graph, based on arbitrary logic.<br>
<br>
DataStream.split() seems to be perfect, except that my source is a CSV file=
, and I have only found built in functions for reading CSV files into a Dat=
aSet.<br>
<br>
I&#39;ve evaluated using DataSet.filter(), but as far as I can tell, that o=
nly allows me to emulate a yes/no split. This is not ideal because it&#39;s=
 too coarse, and I would prefer a more fine grained split than that.<br>
<br>
<br>
Do you have any suggestions on how I can achieve my arbitrary splitting log=
ic for a) DataSets in general, or b) CSV files?<br>
<br>
</blockquote></div><br></div>

--f403045e630c0677fe055bba4935--