flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyula.f...@gmail.com>
Subject Re: Is there Any api that let DataStream join DataSet ?
Date Mon, 29 Jun 2015 09:45:43 GMT
You are right, one cannot use the current window-join implementation to
this.

A workaround is to implement your custom binary stream operator that will
wait until it receives the whole file, then starts joining.
For instance a filestream.connect(streamToJoinWith).flatMap(
CustomCoFlatMap that does the join )

Matthias J. Sax <mjsax@informatik.hu-berlin.de> ezt írta (időpont: 2015.
jún. 29., H, 11:40):

> I am wondering what the semantics of a DataStream created from a file
> is. It should be a regular (but finite) stream. From my understanding, a
> Window-Join is defined with some ts-constraint. So the static file part
> will also have this restriction in the join, right? However, a
> file-stream-join should join *all* data from the file with each element
> in the stream... It seems to me, that a file-DataStream would not yield
> this result. Am I wrong?
>
>
> On 06/29/2015 11:00 AM, Stephan Ewen wrote:
> > If you only want to "join" a finite data set (like a file) to a stream,
> you
> > can do that. you can create a DataStream from a (distributed) file.
> >
> > If you want specific batch-api operations, this is still on the roadmap,
> > not in yet, as Marton said.
> >
> > On Sun, Jun 28, 2015 at 10:45 AM, Márton Balassi <
> balassi.marton@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Flink currently does not have explicit Api support for that, but is
> >> definitely possible to do. In fact Gyula (cc-d) mocked up a prototype
> for a
> >> similar problem some time ago.
> >>
> >> The idea needs some refinement to properly support all the viable use
> cases
> >> though and the streaming Api currently has some more pressing challenges
> >> than this integration. :)
> >>
> >> It's on our roadmap, but is not an immediate task. Could you tell us
> more
> >> about your use case?
> >>
> >> Best,
> >> Marton
> >> On Jun 28, 2015 8:29 AM, "马国维" <maguowei@outlook.com> wrote:
> >>
> >>> Hi,everyone:
> >>> Is there Any api that let the DataStream join a DataSet ? I have read
> all
> >>> the document But I can't find .
> >>> If Flink now does not have the api, will Flink support it in the
> future ?
> >>>  thanks a lot!
> >>>
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message