flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rong Rong <walter...@gmail.com>
Subject Re: [DISCUSS] Adding a mid-term roadmap to the Flink website
Date Thu, 14 Feb 2019 16:45:44 GMT
Hi Stephan,

Thanks for the clarification, yes I think these issues has already been
discussed in previous mailing list threads [1,2,3].

I also agree that updating the "official" roadmap every release is a very
good idea to avoid frequent update.
One question I might've been a bit confusion is: are we suggesting to keep
one roadmap on the documentation site (e.g. [4]) per release, or simply
just one most up-to-date roadmap in the main website [5] ?
Just like the release notes in every release, the former will probably
provide a good tracker for users to look back at previous roadmaps as well
I am assuming.

Thanks,
Rong

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improvement-to-Flink-Window-Operator-with-Slicing-td25750.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-security-improvements-td21068.html
[3]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Kerberos-Improvement-td25983.html

[4] https://ci.apache.org/projects/flink/flink-docs-release-1.7/
[5] https://flink.apache.org/

On Thu, Feb 14, 2019 at 2:26 AM Stephan Ewen <sewen@apache.org> wrote:

> I think the website is better as well.
>
> I agree with Fabian that the wiki is not so visible, and visibility is the
> main motivation.
> This type of roadmap overview would not be updated by everyone - letting
> committers update the roadmap means the listed threads are actually
> happening at the moment.
>
>
> On Thu, Feb 14, 2019 at 11:14 AM Fabian Hueske <fhueske@gmail.com> wrote:
>
>> Hi,
>>
>> I like the idea of putting the roadmap on the website because it is much
>> more visible (and IMO more credible, obligatory) there.
>> However, I share the concerns about frequent updates.
>>
>> It think it would be great to update the "official" roadmap on the
>> website once per release (-bugfix releases), i.e., every three month.
>> We can use the wiki to collect and draft the roadmap for the next update.
>>
>> Best, Fabian
>>
>>
>> Am Do., 14. Feb. 2019 um 11:03 Uhr schrieb Jeff Zhang <zjffdu@gmail.com>:
>>
>>> Hi Stephan,
>>>
>>> Thanks for this proposal. It is a good idea to track the roadmap. One
>>> suggestion is that it might be better to put it into wiki page first.
>>> Because it is easier to update the roadmap on wiki compared to on flink web
>>> site. And I guess we may need to update the roadmap very often at the
>>> beginning as there's so many discussions and proposals in community
>>> recently. We can move it into flink web site later when we feel it could be
>>> nailed down.
>>>
>>> Stephan Ewen <sewen@apache.org> 于2019年2月14日周四 下午5:44写道:
>>>
>>>> Thanks Jincheng and Rong Rong!
>>>>
>>>> I am not deciding a roadmap and making a call on what features should
>>>> be developed or not. I was only collecting broader issues that are already
>>>> happening or have an active FLIP/design discussion plus committer support.
>>>>
>>>> Do we have that for the suggested issues as well? If yes , we can add
>>>> them (can you point me to the issue/mail-thread), if not, let's try and
>>>> move the discussion forward and add them to the roadmap overview then.
>>>>
>>>> Best,
>>>> Stephan
>>>>
>>>>
>>>> On Wed, Feb 13, 2019 at 6:47 PM Rong Rong <walterddr@gmail.com> wrote:
>>>>
>>>>> Thanks Stephan for the great proposal.
>>>>>
>>>>> This would not only be beneficial for new users but also for
>>>>> contributors to keep track on all upcoming features.
>>>>>
>>>>> I think that better window operator support can also be separately
>>>>> group into its own category, as they affects both future DataStream API
and
>>>>> batch stream unification.
>>>>> can we also include:
>>>>> - OVER aggregate for DataStream API separately as @jincheng suggested.
>>>>> - Improving sliding window operator [1]
>>>>>
>>>>> One more additional suggestion, can we also include a more extendable
>>>>> security module [2,3] @shuyi and I are currently working on?
>>>>> This will significantly improve the usability for Flink in corporate
>>>>> environments where proprietary or 3rd-party security integration is needed.
>>>>>
>>>>> Thanks,
>>>>> Rong
>>>>>
>>>>>
>>>>> [1]
>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improvement-to-Flink-Window-Operator-with-Slicing-td25750.html
>>>>> [2]
>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-security-improvements-td21068.html
>>>>> [3]
>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Kerberos-Improvement-td25983.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 13, 2019 at 3:39 AM jincheng sun <sunjincheng121@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Very excited and thank you for launching such a great discussion,
>>>>>> Stephan !
>>>>>>
>>>>>> Here only a little suggestion that in the Batch Streaming Unification
>>>>>> section, do we need to add an item:
>>>>>>
>>>>>> - Same window operators on bounded/unbounded Table API and DataStream
>>>>>> API
>>>>>> (currently OVER window only exists in SQL/TableAPI, DataStream API
>>>>>> does not yet support)
>>>>>>
>>>>>> Best,
>>>>>> Jincheng
>>>>>>
>>>>>> Stephan Ewen <sewen@apache.org> 于2019年2月13日周三 下午7:21写道:
>>>>>>
>>>>>>> Hi all!
>>>>>>>
>>>>>>> Recently several contributors, committers, and users asked about
>>>>>>> making it more visible in which way the project is currently
going.
>>>>>>>
>>>>>>> Users and developers can track the direction by following the
>>>>>>> discussion threads and JIRA, but due to the mass of discussions
and open
>>>>>>> issues, it is very hard to get a good overall picture.
>>>>>>> Especially for new users and contributors, is is very hard to
get a
>>>>>>> quick overview of the project direction.
>>>>>>>
>>>>>>> To fix this, I suggest to add a brief roadmap summary to the
>>>>>>> homepage. It is a bit of a commitment to keep that roadmap up
to date, but
>>>>>>> I think the benefit for users justifies that.
>>>>>>> The Apache Beam project has added such a roadmap [1]
>>>>>>> <https://beam.apache.org/roadmap/>, which was received
very well by
>>>>>>> the community, I would suggest to follow a similar structure
here.
>>>>>>>
>>>>>>> If the community is in favor of this, I would volunteer to write
a
>>>>>>> first version of such a roadmap. The points I would include are
below.
>>>>>>>
>>>>>>> Best,
>>>>>>> Stephan
>>>>>>>
>>>>>>> [1] https://beam.apache.org/roadmap/
>>>>>>>
>>>>>>> ========================================================
>>>>>>>
>>>>>>> Disclaimer: Apache Flink is not governed or steered by any one
>>>>>>> single entity, but by its community and Project Management Committee
(PMC).
>>>>>>> This is not a authoritative roadmap in the sense of a plan with
a specific
>>>>>>> timeline. Instead, we share our vision for the future and major
initiatives
>>>>>>> that are receiving attention and give users and contributors
an
>>>>>>> understanding what they can look forward to.
>>>>>>>
>>>>>>> *Future Role of Table API and DataStream API*
>>>>>>>   - Table API becomes first class citizen
>>>>>>>   - Table API becomes primary API for analytics use cases
>>>>>>>       * Declarative, automatic optimizations
>>>>>>>       * No manual control over state and timers
>>>>>>>   - DataStream API becomes primary API for applications and data
>>>>>>> pipeline use cases
>>>>>>>       * Physical, user controls data types, no magic or optimizer
>>>>>>>       * Explicit control over state and time
>>>>>>>
>>>>>>> *Batch Streaming Unification*
>>>>>>>   - Table API unification (environments) (FLIP-32)
>>>>>>>   - New unified source interface (FLIP-27)
>>>>>>>   - Runtime operator unification & code reuse between DataStream
/
>>>>>>> Table
>>>>>>>   - Extending Table API to make it convenient API for all analytical
>>>>>>> use cases (easier mix in of UDFs)
>>>>>>>   - Same join operators on bounded/unbounded Table API and
>>>>>>> DataStream API
>>>>>>>
>>>>>>> *Faster Batch (Bounded Streams)*
>>>>>>>   - Much of this comes via Blink contribution/merging
>>>>>>>   - Fine-grained Fault Tolerance on bounded data (Table API)
>>>>>>>   - Batch Scheduling on bounded data (Table API)
>>>>>>>   - External Shuffle Services Support on bounded streams
>>>>>>>   - Caching of intermediate results on bounded data (Table API)
>>>>>>>   - Extending DataStream API to explicitly model bounded streams
>>>>>>> (API breaking)
>>>>>>>   - Add fine fault tolerance, scheduling, caching also to DataStream
>>>>>>> API
>>>>>>>
>>>>>>> *Streaming State Evolution*
>>>>>>>   - Let all built-in serializers support stable evolution
>>>>>>>   - First class support for other evolvable formats (Protobuf,
>>>>>>> Thrift)
>>>>>>>   - Savepoint input/output format to modify / adjust savepoints
>>>>>>>
>>>>>>> *Simpler Event Time Handling*
>>>>>>>   - Event Time Alignment in Sources
>>>>>>>   - Simpler out-of-the box support in sources
>>>>>>>
>>>>>>> *Checkpointing*
>>>>>>>   - Consistency of Side Effects: suspend / end with savepoint
>>>>>>> (FLIP-34)
>>>>>>>   - Failed checkpoints explicitly aborted on TaskManagers (not
only
>>>>>>> on coordinator)
>>>>>>>
>>>>>>> *Automatic scaling (adjusting parallelism)*
>>>>>>>   - Reactive scaling
>>>>>>>   - Active scaling policies
>>>>>>>
>>>>>>> *Kubernetes Integration*
>>>>>>>   - Active Kubernetes Integration (Flink actively manages containers)
>>>>>>>
>>>>>>> *SQL Ecosystem*
>>>>>>>   - Extended Metadata Stores / Catalog / Schema Registries support
>>>>>>>   - DDL support
>>>>>>>   - Integration with Hive Ecosystem
>>>>>>>
>>>>>>> *Simpler Handling of Dependencies*
>>>>>>>   - Scala in the APIs, but not in the core (hide in separate
class
>>>>>>> loader)
>>>>>>>   - Hadoop-free by default
>>>>>>>
>>>>>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>

Mime
View raw message