flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benchao Li <libenc...@gmail.com>
Subject Re: Sorting Bounded Streams
Date Sat, 30 May 2020 07:55:20 GMT
Hi Satyam,

You are correct. Blink planner is built on top of DataStream, both for
batch and streaming.
Hence you cannot transform Table into DataSet if you are using blink

AFAIK, the community is working on the unification of batch and streaming.
And the the unification
will be Table API/SQL on top of BoundedStream. Then the DataSet/DataStream
API will be unified
as BoundedStream API. Hence the DataSet API is not the recommended approach
for the long term.

Satyam Shekhar <satyamshekhar@gmail.com> 于2020年5月30日周六 下午3:34写道:

> Thanks for your reply, Benchao Li.
> While I can use the Blink planner in batch mode, I'd still have to work
> with DataSet. Based on my limited reading it appears to me that DataStream
> is being extended to support both batch and steaming use-cases with the
> `isBounded` method in the StreamTableSource interface. Is that correct?
> Is working with DataSet the recommended approach for the long term? Are
> there any performance implications for that decision?
> Regards,
> Satyam
> On Fri, May 29, 2020 at 9:01 PM Benchao Li <libenchao@gmail.com> wrote:
>> Hi Satyam,
>> Are you using blink planner in streaming mode? AFAIK, blink planner in
>> batch mode can sort on arbitrary columns.
>> Satyam Shekhar <satyamshekhar@gmail.com> 于2020年5月30日周六 上午6:19写道:
>>> Hello,
>>> I am using Flink as the streaming execution engine for building a
>>> low-latency alerting application. The use case also requires ad-hoc
>>> querying on batch data, which I also plan to serve using Flink to avoid the
>>> complexity of maintaining two separate engines.
>>> My current understanding is that Order By operator in Blink planner (on
>>> DataStream) requires time attribute as the primary sort column. This is
>>> quite limiting for ad-hoc querying. It seems I can use the DataSet API to
>>> obtain a globally sorted output on an arbitrary column but that will force
>>> me to use the older Flink planner.
>>> Specifically, I am looking for guidance from the community on the
>>> following questions -
>>>    1. Is it possible to obtain a globally sorted output on DataStreams
>>>    on an arbitrary sort column?
>>>    2. What are the tradeoffs in using DataSet vs DataStream in
>>>    performance, long term support, etc?
>>>    3. Is there any other way to address this issue?
>>> Regards,
>>> Satyam
>> --
>> Best,
>> Benchao Li


Benchao Li

View raw message