flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Bucketing Sink does not complete files, when source is from a collection
Date Wed, 04 Apr 2018 20:55:09 GMT
Hi Josh,

You are right, FLINK-2646 is related to the problem of non-finialized
files. If we could distinguish the cases why close() is called, we could do
a proper clean-up if the job terminated because all data was processed.

Right now, the source and sink interfaces of the DataStream API are not
really designed for finite / bounded data.
In order to improve the support for bounded and unbounded data, we have
some plans to design unified interfaces that can handle both cases well.
This effort should also solve cases like the one that you described.

Best, Fabian

2018-04-04 21:51 GMT+02:00 joshlemer <joshlemer@gmail.com>:

> Actually sorry, I have found that this is most likely a manifestation of
> https://issues.apache.org/jira/browse/FLINK-2646 as discussed elsewhere on
> the mailing list. That is, in the second example "fromCollection" the
> entire
> stream ends before a checkpoint is made. Let's hope this is fixed some day
> :-)
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/

View raw message