Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <CAGco--ZtQKC2T3ujAa3dKYL7D-KymO_XL9WXFHCXWq7ET551jQ@mail.gmail.com>
References: <CAGr9p8C3r3JrYNO2FDgOTM+JtFQxSYZsCJoni+2C0_LM1xnAAQ@mail.gmail.com>
 <CALuGr6aFWUenRKfQ0VU5_fpDjVUF_Xwn2VV5Qb1J1EffdwrVfA@mail.gmail.com>
 <CAGco--acM3-6nN=i5H7+FU7bnyuH39US9+asgsvuJRgT2MZxaQ@mail.gmail.com>
 <CALuGr6Ziu56fHSFJ20Xn9YxHacs9AHrN43e2FBGE7mxv0ji59A@mail.gmail.com>
 <CAGr9p8DFEGKBwdVF7X89EY+p1Hxxrm+-7RZLrEwL6cALTOELjQ@mail.gmail.com>
 <CAKiyyaEmBo4ChgqN-qjOoE+wuv1+NafgV7d8XTDrJWUio7HX5g@mail.gmail.com>
 <CAGr9p8BQUGJvjNmDa4CD9Keb8h=AqtkG40NoL9VfV4Ec=wVxFQ@mail.gmail.com>
 <CA+3U0Zmv258dtG1d-bnthGZ_0qO07MUNVoMHPGwNZockfR+g8Q@mail.gmail.com>
 <577637ED.2090702@apache.org> <CAGco--Z3ziDkSLuhEjy2o=_U+DApxHg+Kn7ktky6HOmEiRY+yA@mail.gmail.com>
 <CAGco--ZtQKC2T3ujAa3dKYL7D-KymO_XL9WXFHCXWq7ET551jQ@mail.gmail.com>
From: Maximilian Michels <mxm@apache.org>
Date: Fri, 1 Jul 2016 19:14:21 +0200
Message-ID: <CAGco--Y6wPpFPvbbQrsW1iYqweLEDcj6y6O0AkMuM4JVQ0zi2g@mail.gmail.com>
Subject: Re: [DISCUSS] Releasing Flink 1.1.0
To: dev@flink.apache.org
Content-Type: text/plain; charset=UTF-8
archived-at: Fri, 01 Jul 2016 17:14:44 -0000

Yet another one for the release: FLINK-4144
https://github.com/apache/flink/pull/2191

On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mxm@apache.org> wrote:
> There is also FLINK-4141. We just found this during testing. PR is
> waiting to be merged here: https://github.com/apache/flink/pull/2190
>
> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mxm@apache.org> wrote:
>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
>> and actually a bit tricky to fix. I've addressed the other issues and
>> merged all pending pull requests. Good to go from my side.
>>
>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <chesnay@apache.org> wrote:
>>> There are also 2 fixes for Cassandra that  should be included:
>>> https://github.com/apache/flink/pull/2167
>>> https://github.com/apache/flink/pull/2183
>>>
>>> We should also include the documentation for the metrics stuff (hopefully
>>> merged today)
>>> https://github.com/apache/flink/pull/2158
>>>
>>> In regards to metrics: To add a counter metric a user currently has to call
>>> "counter(...)" on
>>> a MetricGroup. The point was raised in the documentation PR that we may want
>>> to give
>>> them a more descriptive name like "addCounter(...)".
>>>
>>> I would be in favor of changing them but would like others to weigh in on
>>> this. IMO we
>>> should nail this down before 1.1 .
>>>
>>> Regards,
>>> Chesnay
>>>
>>>
>>> On 30.06.2016 22:59, Greg Hogan wrote:
>>>>
>>>> It would be great if hash-based combine (FLINK-3477) could make it in to
>>>> be
>>>> tested for this release. We've seen impressive improvements in performance
>>>> (though, admittedly, some sort-based enhancements are yet to be worked
>>>> on).
>>>> This PR looks to be ripe.
>>>>
>>>> Also, as we tidy up a few things with Gelly and documentation, what is the
>>>> schedule for a feature freeze and creating a 1.1 branch off master?
>>>>
>>>> Thanks,
>>>> Greg
>>>>
>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <rmetzger@apache.org>
>>>> wrote:
>>>>
>>>>> Sure Ufuk! Thanks a lot for taking care of the release management.
>>>>> I'll be on vacation in three weeks, for three weeks and I'm not sure if
>>>>> we
>>>>> get the release done until then.
>>>>>
>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uce@apache.org> wrote:
>>>>>
>>>>>> I would like to do it if that's OK with you Robert. I would follow
>>>>>> your suggestion and wait a few days until the following important
>>>>>> fixes are in:
>>>>>> - Savepoint headers and proper disposal (FLINK-4067 and
>>>>>> https://github.com/apache/flink/pull/2083)
>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
>>>>>> - Table API time support (https://github.com/apache/flink/pull/2150)
>>>>>> - Kafka at-least-once Producer (
>>>>>
>>>>> https://github.com/apache/flink/pull/2108)
>>>>>>
>>>>>> - Cassandra connector fixes (https://github.com/apache/flink/pull/2163)
>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is there something
>>>>>
>>>>> else?)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <rmetzger@apache.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> it doesn't seem that there are volunteers for the RM, so I'll probably
>>>>>
>>>>> do
>>>>>>>
>>>>>>> it.
>>>>>>>
>>>>>>> I try to do the first release candidate (mostly for testing) next week
>>>>>>
>>>>>> (it
>>>>>>>
>>>>>>> depends on the JIRAs fixed by then)
>>>>>>>
>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra <
>>>>>
>>>>> henry.saputra@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the reply, @Max. I was not aware it was about dynamic
>>>>>>
>>>>>> scaling,
>>>>>>>>
>>>>>>>> which I think also asked for YARN support.
>>>>>>>> I agree to list all related half merge JIRA for the ResourceManager.
>>>>>>>>
>>>>>>>> Looking forward for the Apache Mesos integration design for sure =)
>>>>>>>>
>>>>>>>> - Henry
>>>>>>>>
>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian Michels <mxm@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Robert, hi Henry,
>>>>>>>>>
>>>>>>>>> +1 for a 1.1.0 release soon! We have enough new features that
>>>>>
>>>>> justify
>>>>>>>>>
>>>>>>>>> a major release.
>>>>>>>>>
>>>>>>>>> @Henry We have plans to extend the ResourceManager to interact with
>>>>>>>>> the Scheduler which will be a prerequisite for dynamic scaling. I
>>>>>>>>> think this is out of scope for 1.1.0. The upcoming Mesos integration
>>>>>>>>> won't require additional refactoring of the ResourceManager.
>>>>>
>>>>> Instead,
>>>>>>>>>
>>>>>>>>> we will create a new "Dispatcher" component that takes care of
>>>>>>>>> bootstrapping the initial node with the JobManager/ResourceManager.
>>>>>>>>>  From there on, everything will be handled by the Mesos
>>>>>>>>> ResourceManager. I recently discussed this with Eron (CC) who came
>>>>>
>>>>> up
>>>>>>>>>
>>>>>>>>> with this design and he plans to publish it to the mailing list
>>>>>
>>>>> soon.
>>>>>>>>>
>>>>>>>>> How about listing relevant JIRA issues here? "Half Merged" is kind
>>>>>
>>>>> of
>>>>>>>>>
>>>>>>>>> hard to get for people who are not involved in the different
>>>>>>>>> components.
>>>>>>>>>
>>>>>>>>> The Cassandra adapter seems like a pretty important thing to have
>>>>>
>>>>> for
>>>>>>>>>
>>>>>>>>> the next release. In addition, I would like to merge FLINK-3667 and
>>>>>>>>> FLINK-3937. Robert is doing a review at the moment :) Those are a)
>>>>>>>>> refactoring of the command-line and client classes b) adding
>>>>>>>>> capability to resume cluster programmatically.
>>>>>>>>>
>>>>>>>>> Then we should also have a look at any other critical/major bugs
>>>>>>
>>>>>> listed
>>>>>>>>
>>>>>>>> in
>>>>>>>>>
>>>>>>>>> JIRA.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Max
>>>>>>>>>
>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry Saputra <
>>>>>>
>>>>>> henry.saputra@gmail.com
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Robert,
>>>>>>>>>>
>>>>>>>>>> Thanks for staying the discussion.
>>>>>>>>>>
>>>>>>>>>> Do you know if there any open tasks for the Resource Manager left?
>>>>>>>>>>
>>>>>>>>>> That is probably needed for Mesos integration?
>>>>>>>>>>
>>>>>>>>>> - Henry
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger <
>>>>>>
>>>>>> rmetzger@apache.org>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Flink 1.0.0 was released early March, so three months have passed
>>>>>>
>>>>>> and
>>>>>>>>
>>>>>>>> I
>>>>>>>>>>>
>>>>>>>>>>> think we should start discussing the scope of the next major
>>>>>>
>>>>>> release
>>>>>>>>>>>
>>>>>>>>>>> (1.1.0).
>>>>>>>>>>>
>>>>>>>>>>>  From a high level point of view, we've added the following new
>>>>>>>>
>>>>>>>> features:
>>>>>>>>>>>
>>>>>>>>>>> in master:
>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
>>>>>>>>>>> - The metrics system
>>>>>>>>>>> - Kinesis Connector
>>>>>>>>>>> - Persistent file sources for streaming
>>>>>>>>>>>
>>>>>>>>>>> Half merged:
>>>>>>>>>>> - Resource manager refactoring
>>>>>>>>>>>
>>>>>>>>>>> Unmerged features:
>>>>>>>>>>> - Cassandra connector
>>>>>>>>>>> - Key groups ("rescaling from savepoints")
>>>>>>>>>>> - Queryable state
>>>>>>>>>>>
>>>>>>>>>>> I'm pretty sure I forgot many other features / pull requests,
>>>>>>
>>>>>> please
>>>>>>>>>
>>>>>>>>> post
>>>>>>>>>>>
>>>>>>>>>>> them to this thread. I'll collect them and create a Wiki page out
>>>>>>
>>>>>> of
>>>>>>>>
>>>>>>>> it.
>>>>>>>>>>>
>>>>>>>>>>> Some immediate TODOs for us:
>>>>>>>>>>> - Which of the unmerged features are we going to add to the
>>>>>>
>>>>>> release?
>>>>>>>>>>>
>>>>>>>>>>> - Which blockers do we need to address before releasing?
>>>>>>>>>>> - Are there any volunteers for the release manager?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Robert
>>>>>>>>>>>
>>>