flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: [DISCUSS] Releasing Flink 1.1.0
Date Mon, 04 Jul 2016 19:05:12 GMT
+1 to do a RC0 this week, but the master-forking with RC1. I would like to
reduce the time we need to apply patches to multiple branches.

@Aljoscha: I was running into the same issue on EMR when I used Flink w/
RocksDB recently, so I agree ;)



On Mon, Jul 4, 2016 at 3:35 PM, Aljoscha Krettek <aljoscha@apache.org>
wrote:

> IMHO, the fix for this should also go in:
> https://issues.apache.org/jira/browse/FLINK-4115. This is blocking for
> users that want to use the RocksDB backend or FsStateBackend on Amazon EMR
> with S3.
>
> There is already an open PR that I'm hoping to get in this week.
>
> On Mon, 4 Jul 2016 at 13:48 Ufuk Celebi <uce@apache.org> wrote:
>
> > Thanks for the feedback. I would like to create a preview RC0 this
> > week like we did for the last releases, too. In past major releases,
> > we needed to create many release candidates, often for fixing just
> > some small issues. I would like to speed up the release process by
> > collecting as many issues as possible now with the RC0. Once these
> > issues are resolved, we can start voting with the RC1. This
> > essentially means that we have a feature freeze. I would create the
> > release-1.1 branch with RC1.
> >
> > Regarding the open issues:
> >
> > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > headers (#2194) and the savepoint disposal PR needs addressing of
> > comments.
> >
> > - The Cassandra connector fixes are either merged or very close to be
> > merged.
> >
> > - Metrics docs are not a blocker since the online docs are updated
> > after the release. Regarding the renaming, we should decide soon. Any
> > opinions here?
> >
> > - The YARN issues have been resolved.
> >
> > I will also make a pass over JIRA and our PRs to check whether we've
> > missed something.
> >
> > @Greg: You are right, the hash-based combine PR has been extensively
> > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > look into it and cannot tell whether we should merge it now. Is the
> > hash-based combine strategy activated by default or does the user has
> > to activate it manually? The later case would make me feel more
> > comfortable merging it for the release.
> >
> >
> > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mxm@apache.org>
> wrote:
> > > Yet another one for the release: FLINK-4144
> > > https://github.com/apache/flink/pull/2191
> > >
> > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mxm@apache.org>
> > wrote:
> > >> There is also FLINK-4141. We just found this during testing. PR is
> > >> waiting to be merged here: https://github.com/apache/flink/pull/2190
> > >>
> > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mxm@apache.org>
> > wrote:
> > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this release
> > >>> and actually a bit tricky to fix. I've addressed the other issues and
> > >>> merged all pending pull requests. Good to go from my side.
> > >>>
> > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> chesnay@apache.org>
> > wrote:
> > >>>> There are also 2 fixes for Cassandra that  should be included:
> > >>>> https://github.com/apache/flink/pull/2167
> > >>>> https://github.com/apache/flink/pull/2183
> > >>>>
> > >>>> We should also include the documentation for the metrics stuff
> > (hopefully
> > >>>> merged today)
> > >>>> https://github.com/apache/flink/pull/2158
> > >>>>
> > >>>> In regards to metrics: To add a counter metric a user currently
has
> > to call
> > >>>> "counter(...)" on
> > >>>> a MetricGroup. The point was raised in the documentation PR that
we
> > may want
> > >>>> to give
> > >>>> them a more descriptive name like "addCounter(...)".
> > >>>>
> > >>>> I would be in favor of changing them but would like others to weigh
> > in on
> > >>>> this. IMO we
> > >>>> should nail this down before 1.1 .
> > >>>>
> > >>>> Regards,
> > >>>> Chesnay
> > >>>>
> > >>>>
> > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > >>>>>
> > >>>>> It would be great if hash-based combine (FLINK-3477) could
make it
> > in to
> > >>>>> be
> > >>>>> tested for this release. We've seen impressive improvements
in
> > performance
> > >>>>> (though, admittedly, some sort-based enhancements are yet to
be
> > worked
> > >>>>> on).
> > >>>>> This PR looks to be ripe.
> > >>>>>
> > >>>>> Also, as we tidy up a few things with Gelly and documentation,
what
> > is the
> > >>>>> schedule for a feature freeze and creating a 1.1 branch off
master?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Greg
> > >>>>>
> > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> rmetzger@apache.org
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
management.
> > >>>>>> I'll be on vacation in three weeks, for three weeks and
I'm not
> > sure if
> > >>>>>> we
> > >>>>>> get the release done until then.
> > >>>>>>
> > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uce@apache.org>
> > wrote:
> > >>>>>>
> > >>>>>>> I would like to do it if that's OK with you Robert.
I would
> follow
> > >>>>>>> your suggestion and wait a few days until the following
important
> > >>>>>>> fixes are in:
> > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067
and
> > >>>>>>> https://github.com/apache/flink/pull/2083)
> > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > >>>>>>> - Table API time support (
> > https://github.com/apache/flink/pull/2150)
> > >>>>>>> - Kafka at-least-once Producer (
> > >>>>>>
> > >>>>>> https://github.com/apache/flink/pull/2108)
> > >>>>>>>
> > >>>>>>> - Cassandra connector fixes (
> > https://github.com/apache/flink/pull/2163)
> > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max: is
there
> > something
> > >>>>>>
> > >>>>>> else?)
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger <
> > rmetzger@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>> it doesn't seem that there are volunteers for the
RM, so I'll
> > probably
> > >>>>>>
> > >>>>>> do
> > >>>>>>>>
> > >>>>>>>> it.
> > >>>>>>>>
> > >>>>>>>> I try to do the first release candidate (mostly
for testing)
> next
> > week
> > >>>>>>>
> > >>>>>>> (it
> > >>>>>>>>
> > >>>>>>>> depends on the JIRAs fixed by then)
> > >>>>>>>>
> > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra
<
> > >>>>>>
> > >>>>>> henry.saputra@gmail.com
> > >>>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks for the reply, @Max. I was not aware
it was about
> dynamic
> > >>>>>>>
> > >>>>>>> scaling,
> > >>>>>>>>>
> > >>>>>>>>> which I think also asked for YARN support.
> > >>>>>>>>> I agree to list all related half merge JIRA
for the
> > ResourceManager.
> > >>>>>>>>>
> > >>>>>>>>> Looking forward for the Apache Mesos integration
design for
> sure
> > =)
> > >>>>>>>>>
> > >>>>>>>>> - Henry
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian
Michels <
> > mxm@apache.org>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Robert, hi Henry,
> > >>>>>>>>>>
> > >>>>>>>>>> +1 for a 1.1.0 release soon! We have enough
new features that
> > >>>>>>
> > >>>>>> justify
> > >>>>>>>>>>
> > >>>>>>>>>> a major release.
> > >>>>>>>>>>
> > >>>>>>>>>> @Henry We have plans to extend the ResourceManager
to interact
> > with
> > >>>>>>>>>> the Scheduler which will be a prerequisite
for dynamic
> scaling.
> > I
> > >>>>>>>>>> think this is out of scope for 1.1.0. The
upcoming Mesos
> > integration
> > >>>>>>>>>> won't require additional refactoring of
the ResourceManager.
> > >>>>>>
> > >>>>>> Instead,
> > >>>>>>>>>>
> > >>>>>>>>>> we will create a new "Dispatcher" component
that takes care of
> > >>>>>>>>>> bootstrapping the initial node with the
> > JobManager/ResourceManager.
> > >>>>>>>>>>  From there on, everything will be handled
by the Mesos
> > >>>>>>>>>> ResourceManager. I recently discussed this
with Eron (CC) who
> > came
> > >>>>>>
> > >>>>>> up
> > >>>>>>>>>>
> > >>>>>>>>>> with this design and he plans to publish
it to the mailing
> list
> > >>>>>>
> > >>>>>> soon.
> > >>>>>>>>>>
> > >>>>>>>>>> How about listing relevant JIRA issues
here? "Half Merged" is
> > kind
> > >>>>>>
> > >>>>>> of
> > >>>>>>>>>>
> > >>>>>>>>>> hard to get for people who are not involved
in the different
> > >>>>>>>>>> components.
> > >>>>>>>>>>
> > >>>>>>>>>> The Cassandra adapter seems like a pretty
important thing to
> > have
> > >>>>>>
> > >>>>>> for
> > >>>>>>>>>>
> > >>>>>>>>>> the next release. In addition, I would
like to merge
> FLINK-3667
> > and
> > >>>>>>>>>> FLINK-3937. Robert is doing a review at
the moment :) Those
> are
> > a)
> > >>>>>>>>>> refactoring of the command-line and client
classes b) adding
> > >>>>>>>>>> capability to resume cluster programmatically.
> > >>>>>>>>>>
> > >>>>>>>>>> Then we should also have a look at any
other critical/major
> bugs
> > >>>>>>>
> > >>>>>>> listed
> > >>>>>>>>>
> > >>>>>>>>> in
> > >>>>>>>>>>
> > >>>>>>>>>> JIRA.
> > >>>>>>>>>>
> > >>>>>>>>>> Cheers,
> > >>>>>>>>>> Max
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM, Henry
Saputra <
> > >>>>>>>
> > >>>>>>> henry.saputra@gmail.com
> > >>>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi Robert,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for staying the discussion.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Do you know if there any open tasks
for the Resource Manager
> > left?
> > >>>>>>>>>>>
> > >>>>>>>>>>> That is probably needed for Mesos integration?
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Henry
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55 PM, Robert
Metzger <
> > >>>>>>>
> > >>>>>>> rmetzger@apache.org>
> > >>>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Flink 1.0.0 was released early
March, so three months have
> > passed
> > >>>>>>>
> > >>>>>>> and
> > >>>>>>>>>
> > >>>>>>>>> I
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> think we should start discussing
the scope of the next major
> > >>>>>>>
> > >>>>>>> release
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> (1.1.0).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  From a high level point of view,
we've added the following
> > new
> > >>>>>>>>>
> > >>>>>>>>> features:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> in master:
> > >>>>>>>>>>>> - Table API Refactoring, SQL, StreamSQL
> > >>>>>>>>>>>> - The metrics system
> > >>>>>>>>>>>> - Kinesis Connector
> > >>>>>>>>>>>> - Persistent file sources for streaming
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Half merged:
> > >>>>>>>>>>>> - Resource manager refactoring
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Unmerged features:
> > >>>>>>>>>>>> - Cassandra connector
> > >>>>>>>>>>>> - Key groups ("rescaling from savepoints")
> > >>>>>>>>>>>> - Queryable state
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'm pretty sure I forgot many other
features / pull
> requests,
> > >>>>>>>
> > >>>>>>> please
> > >>>>>>>>>>
> > >>>>>>>>>> post
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> them to this thread. I'll collect
them and create a Wiki
> page
> > out
> > >>>>>>>
> > >>>>>>> of
> > >>>>>>>>>
> > >>>>>>>>> it.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Some immediate TODOs for us:
> > >>>>>>>>>>>> - Which of the unmerged features
are we going to add to the
> > >>>>>>>
> > >>>>>>> release?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - Which blockers do we need to
address before releasing?
> > >>>>>>>>>>>> - Are there any volunteers for
the release manager?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regards,
> > >>>>>>>>>>>> Robert
> > >>>>>>>>>>>>
> > >>>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message