flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: [DISCUSS] Releasing Flink 1.1.0
Date Tue, 05 Jul 2016 15:32:15 GMT
Btw, another blocking issue, IMHO:
https://issues.apache.org/jira/browse/FLINK-4149

I'm working on a fix.


On Tue, 5 Jul 2016 at 17:08 Till Rohrmann <trohrmann@apache.org> wrote:

> I found another critical issue [1]. The murmur hash correction introduced
> between Flink 1.0 and 1.1 breaks the backwards compatibility with respect
> to savepoints. I think we have to fix this for the release.
>
> @Ufuk, I'm not sure whether I find time this week to work on FLINK-4150. I
> could make it a stretch goal for this sprint. Or we schedule it in next
> week's sprint.
>
> [1] https://issues.apache.org/jira/browse/FLINK-4154
>
>
>
> On Tue, Jul 5, 2016 at 3:56 PM, Greg Hogan <code@greghogan.com> wrote:
>
> > Hi Ufuk,
> >
> > The old sort-based combine is still the default. The user calls
> > .setCombineHint(CombineHint) to make a selection (I think this was
> > originally overloaded on DataSet and it looks like the pr1517
> documentation
> > update does not reflect the new usage).
> >
> > I'd be glad to merge this in but I didn't want to preempt Gabor or
> Fabian.
> >
> > Greg
> >
> > On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uce@apache.org> wrote:
> >
> > > Thanks for the feedback. I would like to create a preview RC0 this
> > > week like we did for the last releases, too. In past major releases,
> > > we needed to create many release candidates, often for fixing just
> > > some small issues. I would like to speed up the release process by
> > > collecting as many issues as possible now with the RC0. Once these
> > > issues are resolved, we can start voting with the RC1. This
> > > essentially means that we have a feature freeze. I would create the
> > > release-1.1 branch with RC1.
> > >
> > > Regarding the open issues:
> > >
> > > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > > headers (#2194) and the savepoint disposal PR needs addressing of
> > > comments.
> > >
> > > - The Cassandra connector fixes are either merged or very close to be
> > > merged.
> > >
> > > - Metrics docs are not a blocker since the online docs are updated
> > > after the release. Regarding the renaming, we should decide soon. Any
> > > opinions here?
> > >
> > > - The YARN issues have been resolved.
> > >
> > > I will also make a pass over JIRA and our PRs to check whether we've
> > > missed something.
> > >
> > > @Greg: You are right, the hash-based combine PR has been extensively
> > > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > > look into it and cannot tell whether we should merge it now. Is the
> > > hash-based combine strategy activated by default or does the user has
> > > to activate it manually? The later case would make me feel more
> > > comfortable merging it for the release.
> > >
> > >
> > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mxm@apache.org>
> > wrote:
> > > > Yet another one for the release: FLINK-4144
> > > > https://github.com/apache/flink/pull/2191
> > > >
> > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mxm@apache.org>
> > > wrote:
> > > >> There is also FLINK-4141. We just found this during testing. PR is
> > > >> waiting to be merged here:
> https://github.com/apache/flink/pull/2190
> > > >>
> > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mxm@apache.org>
> > > wrote:
> > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this
> release
> > > >>> and actually a bit tricky to fix. I've addressed the other issues
> and
> > > >>> merged all pending pull requests. Good to go from my side.
> > > >>>
> > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> > chesnay@apache.org>
> > > wrote:
> > > >>>> There are also 2 fixes for Cassandra that  should be included:
> > > >>>> https://github.com/apache/flink/pull/2167
> > > >>>> https://github.com/apache/flink/pull/2183
> > > >>>>
> > > >>>> We should also include the documentation for the metrics stuff
> > > (hopefully
> > > >>>> merged today)
> > > >>>> https://github.com/apache/flink/pull/2158
> > > >>>>
> > > >>>> In regards to metrics: To add a counter metric a user currently
> has
> > > to call
> > > >>>> "counter(...)" on
> > > >>>> a MetricGroup. The point was raised in the documentation PR
that
> we
> > > may want
> > > >>>> to give
> > > >>>> them a more descriptive name like "addCounter(...)".
> > > >>>>
> > > >>>> I would be in favor of changing them but would like others
to
> weigh
> > > in on
> > > >>>> this. IMO we
> > > >>>> should nail this down before 1.1 .
> > > >>>>
> > > >>>> Regards,
> > > >>>> Chesnay
> > > >>>>
> > > >>>>
> > > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > > >>>>>
> > > >>>>> It would be great if hash-based combine (FLINK-3477) could
make
> it
> > > in to
> > > >>>>> be
> > > >>>>> tested for this release. We've seen impressive improvements
in
> > > performance
> > > >>>>> (though, admittedly, some sort-based enhancements are
yet to be
> > > worked
> > > >>>>> on).
> > > >>>>> This PR looks to be ripe.
> > > >>>>>
> > > >>>>> Also, as we tidy up a few things with Gelly and documentation,
> what
> > > is the
> > > >>>>> schedule for a feature freeze and creating a 1.1 branch
off
> master?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Greg
> > > >>>>>
> > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> > rmetzger@apache.org
> > > >
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
> management.
> > > >>>>>> I'll be on vacation in three weeks, for three weeks
and I'm not
> > > sure if
> > > >>>>>> we
> > > >>>>>> get the release done until then.
> > > >>>>>>
> > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uce@apache.org>
> > > wrote:
> > > >>>>>>
> > > >>>>>>> I would like to do it if that's OK with you Robert.
I would
> > follow
> > > >>>>>>> your suggestion and wait a few days until the
following
> important
> > > >>>>>>> fixes are in:
> > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067
and
> > > >>>>>>> https://github.com/apache/flink/pull/2083)
> > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > > >>>>>>> - Table API time support (
> > > https://github.com/apache/flink/pull/2150)
> > > >>>>>>> - Kafka at-least-once Producer (
> > > >>>>>>
> > > >>>>>> https://github.com/apache/flink/pull/2108)
> > > >>>>>>>
> > > >>>>>>> - Cassandra connector fixes (
> > > https://github.com/apache/flink/pull/2163)
> > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max:
is there
> > > something
> > > >>>>>>
> > > >>>>>> else?)
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger
<
> > > rmetzger@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Hi,
> > > >>>>>>>> it doesn't seem that there are volunteers
for the RM, so I'll
> > > probably
> > > >>>>>>
> > > >>>>>> do
> > > >>>>>>>>
> > > >>>>>>>> it.
> > > >>>>>>>>
> > > >>>>>>>> I try to do the first release candidate (mostly
for testing)
> > next
> > > week
> > > >>>>>>>
> > > >>>>>>> (it
> > > >>>>>>>>
> > > >>>>>>>> depends on the JIRAs fixed by then)
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra
<
> > > >>>>>>
> > > >>>>>> henry.saputra@gmail.com
> > > >>>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thanks for the reply, @Max. I was not
aware it was about
> > dynamic
> > > >>>>>>>
> > > >>>>>>> scaling,
> > > >>>>>>>>>
> > > >>>>>>>>> which I think also asked for YARN support.
> > > >>>>>>>>> I agree to list all related half merge
JIRA for the
> > > ResourceManager.
> > > >>>>>>>>>
> > > >>>>>>>>> Looking forward for the Apache Mesos integration
design for
> > sure
> > > =)
> > > >>>>>>>>>
> > > >>>>>>>>> - Henry
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian
Michels <
> > > mxm@apache.org>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Robert, hi Henry,
> > > >>>>>>>>>>
> > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have
enough new features
> that
> > > >>>>>>
> > > >>>>>> justify
> > > >>>>>>>>>>
> > > >>>>>>>>>> a major release.
> > > >>>>>>>>>>
> > > >>>>>>>>>> @Henry We have plans to extend the
ResourceManager to
> interact
> > > with
> > > >>>>>>>>>> the Scheduler which will be a prerequisite
for dynamic
> > scaling.
> > > I
> > > >>>>>>>>>> think this is out of scope for 1.1.0.
The upcoming Mesos
> > > integration
> > > >>>>>>>>>> won't require additional refactoring
of the ResourceManager.
> > > >>>>>>
> > > >>>>>> Instead,
> > > >>>>>>>>>>
> > > >>>>>>>>>> we will create a new "Dispatcher"
component that takes care
> of
> > > >>>>>>>>>> bootstrapping the initial node with
the
> > > JobManager/ResourceManager.
> > > >>>>>>>>>>  From there on, everything will be
handled by the Mesos
> > > >>>>>>>>>> ResourceManager. I recently discussed
this with Eron (CC)
> who
> > > came
> > > >>>>>>
> > > >>>>>> up
> > > >>>>>>>>>>
> > > >>>>>>>>>> with this design and he plans to publish
it to the mailing
> > list
> > > >>>>>>
> > > >>>>>> soon.
> > > >>>>>>>>>>
> > > >>>>>>>>>> How about listing relevant JIRA issues
here? "Half Merged"
> is
> > > kind
> > > >>>>>>
> > > >>>>>> of
> > > >>>>>>>>>>
> > > >>>>>>>>>> hard to get for people who are not
involved in the different
> > > >>>>>>>>>> components.
> > > >>>>>>>>>>
> > > >>>>>>>>>> The Cassandra adapter seems like a
pretty important thing to
> > > have
> > > >>>>>>
> > > >>>>>> for
> > > >>>>>>>>>>
> > > >>>>>>>>>> the next release. In addition, I would
like to merge
> > FLINK-3667
> > > and
> > > >>>>>>>>>> FLINK-3937. Robert is doing a review
at the moment :) Those
> > are
> > > a)
> > > >>>>>>>>>> refactoring of the command-line and
client classes b) adding
> > > >>>>>>>>>> capability to resume cluster programmatically.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Then we should also have a look at
any other critical/major
> > bugs
> > > >>>>>>>
> > > >>>>>>> listed
> > > >>>>>>>>>
> > > >>>>>>>>> in
> > > >>>>>>>>>>
> > > >>>>>>>>>> JIRA.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Cheers,
> > > >>>>>>>>>> Max
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM,
Henry Saputra <
> > > >>>>>>>
> > > >>>>>>> henry.saputra@gmail.com
> > > >>>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hi Robert,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for staying the discussion.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Do you know if there any open
tasks for the Resource
> Manager
> > > left?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That is probably needed for Mesos
integration?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Henry
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55
PM, Robert Metzger <
> > > >>>>>>>
> > > >>>>>>> rmetzger@apache.org>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Flink 1.0.0 was released early
March, so three months have
> > > passed
> > > >>>>>>>
> > > >>>>>>> and
> > > >>>>>>>>>
> > > >>>>>>>>> I
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> think we should start discussing
the scope of the next
> major
> > > >>>>>>>
> > > >>>>>>> release
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> (1.1.0).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>  From a high level point of
view, we've added the
> following
> > > new
> > > >>>>>>>>>
> > > >>>>>>>>> features:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> in master:
> > > >>>>>>>>>>>> - Table API Refactoring, SQL,
StreamSQL
> > > >>>>>>>>>>>> - The metrics system
> > > >>>>>>>>>>>> - Kinesis Connector
> > > >>>>>>>>>>>> - Persistent file sources
for streaming
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Half merged:
> > > >>>>>>>>>>>> - Resource manager refactoring
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Unmerged features:
> > > >>>>>>>>>>>> - Cassandra connector
> > > >>>>>>>>>>>> - Key groups ("rescaling from
savepoints")
> > > >>>>>>>>>>>> - Queryable state
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I'm pretty sure I forgot many
other features / pull
> > requests,
> > > >>>>>>>
> > > >>>>>>> please
> > > >>>>>>>>>>
> > > >>>>>>>>>> post
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> them to this thread. I'll
collect them and create a Wiki
> > page
> > > out
> > > >>>>>>>
> > > >>>>>>> of
> > > >>>>>>>>>
> > > >>>>>>>>> it.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Some immediate TODOs for us:
> > > >>>>>>>>>>>> - Which of the unmerged features
are we going to add to
> the
> > > >>>>>>>
> > > >>>>>>> release?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - Which blockers do we need
to address before releasing?
> > > >>>>>>>>>>>> - Are there any volunteers
for the release manager?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>> Robert
> > > >>>>>>>>>>>>
> > > >>>>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message