flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: [DISCUSS] Releasing Flink 1.1.0
Date Mon, 04 Jul 2016 11:47:54 GMT
Thanks for the feedback. I would like to create a preview RC0 this
week like we did for the last releases, too. In past major releases,
we needed to create many release candidates, often for fixing just
some small issues. I would like to speed up the release process by
collecting as many issues as possible now with the RC0. Once these
issues are resolved, we can start voting with the RC1. This
essentially means that we have a feature freeze. I would create the
release-1.1 branch with RC1.

Regarding the open issues:

- The savepoint fixes are not yet in. There is a PR for the savepoint
headers (#2194) and the savepoint disposal PR needs addressing of
comments.

- The Cassandra connector fixes are either merged or very close to be merged.

- Metrics docs are not a blocker since the online docs are updated
after the release. Regarding the renaming, we should decide soon. Any
opinions here?

- The YARN issues have been resolved.

I will also make a pass over JIRA and our PRs to check whether we've
missed something.

@Greg: You are right, the hash-based combine PR has been extensively
reviewed. Unfortunately, I'm not familiar with the code as I didn't
look into it and cannot tell whether we should merge it now. Is the
hash-based combine strategy activated by default or does the user has
to activate it manually? The later case would make me feel more
comfortable merging it for the release.


On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mxm@apache.org> wrote:
> Yet another one for the release: FLINK-4144
> https://github.com/apache/flink/pull/2191
>
> On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mxm@apache.org> wrote:
>> There is also FLINK-4141. We just found this during testing. PR is
>> waiting to be merged here: https://github.com/apache/flink/pull/2190
>>
>> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mxm@apache.org> wrote:
>>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
>>> and actually a bit tricky to fix. I've addressed the other issues and
>>> merged all pending pull requests. Good to go from my side.
>>>
>>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <chesnay@apache.org>
wrote:
>>>> There are also 2 fixes for Cassandra that  should be included:
>>>> https://github.com/apache/flink/pull/2167
>>>> https://github.com/apache/flink/pull/2183
>>>>
>>>> We should also include the documentation for the metrics stuff (hopefully
>>>> merged today)
>>>> https://github.com/apache/flink/pull/2158
>>>>
>>>> In regards to metrics: To add a counter metric a user currently has to call
>>>> "counter(...)" on
>>>> a MetricGroup. The point was raised in the documentation PR that we may want
>>>> to give
>>>> them a more descriptive name like "addCounter(...)".
>>>>
>>>> I would be in favor of changing them but would like others to weigh in on
>>>> this. IMO we
>>>> should nail this down before 1.1 .
>>>>
>>>> Regards,
>>>> Chesnay
>>>>
>>>>
>>>> On 30.06.2016 22:59, Greg Hogan wrote:
>>>>>
>>>>> It would be great if hash-based combine (FLINK-3477) could make it in
to
>>>>> be
>>>>> tested for this release. We've seen impressive improvements in performance
>>>>> (though, admittedly, some sort-based enhancements are yet to be worked
>>>>> on).
>>>>> This PR looks to be ripe.
>>>>>
>>>>> Also, as we tidy up a few things with Gelly and documentation, what is
the
>>>>> schedule for a feature freeze and creating a 1.1 branch off master?
>>>>>
>>>>> Thanks,
>>>>> Greg
>>>>>
>>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rmetzger@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Sure Ufuk! Thanks a lot for taking care of the release management.
>>>>>> I'll be on vacation in three weeks, for three weeks and I'm not sure
if
>>>>>> we
>>>>>> get the release done until then.
>>>>>>
>>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uce@apache.org>
wrote:
>>>>>>
>>>>>>> I would like to do it if that's OK with you Robert. I would follow
>>>>>>> your suggestion and wait a few days until the following important
>>>>>>> fixes are in:
>>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
>>>>>>> https://github.com/apache/flink/pull/2083)
>>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
>>>>>>> - Table API time support (https://github.com/apache/flink/pull/2150)
>>>>>>> - Kafka at-least-once Producer (
>>>>>>
>>>>>> https://github.com/apache/flink/pull/2108)
>>>>>>>
>>>>>>> - Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
>>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something
>>>>>>
>>>>>> else?)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rmetzger@apache.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> it doesn't seem that there are volunteers for the RM, so
I'll probably
>>>>>>
>>>>>> do
>>>>>>>>
>>>>>>>> it.
>>>>>>>>
>>>>>>>> I try to do the first release candidate (mostly for testing)
next week
>>>>>>>
>>>>>>> (it
>>>>>>>>
>>>>>>>> depends on the JIRAs fixed by then)
>>>>>>>>
>>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
>>>>>>
>>>>>> henry.saputra@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the reply, @Max. I was not aware it was about
dynamic
>>>>>>>
>>>>>>> scaling,
>>>>>>>>>
>>>>>>>>> which I think also asked for YARN support.
>>>>>>>>> I agree to list all related half merge JIRA for the ResourceManager.
>>>>>>>>>
>>>>>>>>> Looking forward for the Apache Mesos integration design
for sure =)
>>>>>>>>>
>>>>>>>>> - Henry
>>>>>>>>>
>>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mxm@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Robert, hi Henry,
>>>>>>>>>>
>>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features
that
>>>>>>
>>>>>> justify
>>>>>>>>>>
>>>>>>>>>> a major release.
>>>>>>>>>>
>>>>>>>>>> @Henry We have plans to extend the ResourceManager
to interact with
>>>>>>>>>> the Scheduler which will be a prerequisite for dynamic
scaling. I
>>>>>>>>>> think this is out of scope for 1.1.0. The upcoming
Mesos integration
>>>>>>>>>> won't require additional refactoring of the ResourceManager.
>>>>>>
>>>>>> Instead,
>>>>>>>>>>
>>>>>>>>>> we will create a new "Dispatcher" component that
takes care of
>>>>>>>>>> bootstrapping the initial node with the JobManager/ResourceManager.
>>>>>>>>>>  From there on, everything will be handled by the
Mesos
>>>>>>>>>> ResourceManager. I recently discussed this with Eron
(CC) who came
>>>>>>
>>>>>> up
>>>>>>>>>>
>>>>>>>>>> with this design and he plans to publish it to the
mailing list
>>>>>>
>>>>>> soon.
>>>>>>>>>>
>>>>>>>>>> How about listing relevant JIRA issues here? "Half
Merged" is kind
>>>>>>
>>>>>> of
>>>>>>>>>>
>>>>>>>>>> hard to get for people who are not involved in the
different
>>>>>>>>>> components.
>>>>>>>>>>
>>>>>>>>>> The Cassandra adapter seems like a pretty important
thing to have
>>>>>>
>>>>>> for
>>>>>>>>>>
>>>>>>>>>> the next release. In addition, I would like to merge
FLINK-3667 and
>>>>>>>>>> FLINK-3937. Robert is doing a review at the moment
:) Those are a)
>>>>>>>>>> refactoring of the command-line and client classes
b) adding
>>>>>>>>>> capability to resume cluster programmatically.
>>>>>>>>>>
>>>>>>>>>> Then we should also have a look at any other critical/major
bugs
>>>>>>>
>>>>>>> listed
>>>>>>>>>
>>>>>>>>> in
>>>>>>>>>>
>>>>>>>>>> JIRA.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Max
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
>>>>>>>
>>>>>>> henry.saputra@gmail.com
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Robert,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for staying the discussion.
>>>>>>>>>>>
>>>>>>>>>>> Do you know if there any open tasks for the Resource
Manager left?
>>>>>>>>>>>
>>>>>>>>>>> That is probably needed for Mesos integration?
>>>>>>>>>>>
>>>>>>>>>>> - Henry
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger
<
>>>>>>>
>>>>>>> rmetzger@apache.org>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Flink 1.0.0 was released early March, so
three months have passed
>>>>>>>
>>>>>>> and
>>>>>>>>>
>>>>>>>>> I
>>>>>>>>>>>>
>>>>>>>>>>>> think we should start discussing the scope
of the next major
>>>>>>>
>>>>>>> release
>>>>>>>>>>>>
>>>>>>>>>>>> (1.1.0).
>>>>>>>>>>>>
>>>>>>>>>>>>  From a high level point of view, we've added
the following new
>>>>>>>>>
>>>>>>>>> features:
>>>>>>>>>>>>
>>>>>>>>>>>> in master:
>>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
>>>>>>>>>>>> - The metrics system
>>>>>>>>>>>> - Kinesis Connector
>>>>>>>>>>>> - Persistent file sources for streaming
>>>>>>>>>>>>
>>>>>>>>>>>> Half merged:
>>>>>>>>>>>> - Resource manager refactoring
>>>>>>>>>>>>
>>>>>>>>>>>> Unmerged features:
>>>>>>>>>>>> - Cassandra connector
>>>>>>>>>>>> - Key groups ("rescaling from savepoints")
>>>>>>>>>>>> - Queryable state
>>>>>>>>>>>>
>>>>>>>>>>>> I'm pretty sure I forgot many other features
/ pull requests,
>>>>>>>
>>>>>>> please
>>>>>>>>>>
>>>>>>>>>> post
>>>>>>>>>>>>
>>>>>>>>>>>> them to this thread. I'll collect them and
create a Wiki page out
>>>>>>>
>>>>>>> of
>>>>>>>>>
>>>>>>>>> it.
>>>>>>>>>>>>
>>>>>>>>>>>> Some immediate TODOs for us:
>>>>>>>>>>>> - Which of the unmerged features are we going
to add to the
>>>>>>>
>>>>>>> release?
>>>>>>>>>>>>
>>>>>>>>>>>> - Which blockers do we need to address before
releasing?
>>>>>>>>>>>> - Are there any volunteers for the release
manager?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Robert
>>>>>>>>>>>>
>>>>

Mime
View raw message