flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: [DISCUSS] Releasing Flink 1.1.0
Date Tue, 05 Jul 2016 16:18:47 GMT
it's just because Fabian said that it's better to not mix java and scala
(as you can see in the comments of that PR)
On 5 Jul 2016 18:53, "Aljoscha Krettek" <aljoscha@apache.org> wrote:

> I think it's not strictly required that all code be in Scala. There is
> already some Java code in there so we shouldn't force people to write Scala
> code if they make a valuable contribution in Java.
>
> On Tue, 5 Jul 2016 at 17:33 Flavio Pompermaier <pompermaier@okkam.it>
> wrote:
>
> > Hi to all,
> > if Flink 1.1 will introduce ufficially the Table API, do you think
> someone
> > could take care of rewriting in scala the necessary java code of my PR
> > about reading CSV as Rows instead of tuples[1]?
> >
> > For our use cases, and many new users approaching to Flink IMHO, that
> will
> > be definitely useful because you can read csv files with more than 25
> > columns and properly handle null values. What  do you think?
> >
> > [1] https://github.com/apache/flink/pull/1989
> >
> > Best,
> > Flavio
> > On 5 Jul 2016 18:08, "Till Rohrmann" <trohrmann@apache.org> wrote:
> >
> > I found another critical issue [1]. The murmur hash correction introduced
> > between Flink 1.0 and 1.1 breaks the backwards compatibility with respect
> > to savepoints. I think we have to fix this for the release.
> >
> > @Ufuk, I'm not sure whether I find time this week to work on FLINK-4150.
> I
> > could make it a stretch goal for this sprint. Or we schedule it in next
> > week's sprint.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-4154
> >
> >
> >
> > On Tue, Jul 5, 2016 at 3:56 PM, Greg Hogan <code@greghogan.com> wrote:
> >
> > > Hi Ufuk,
> > >
> > > The old sort-based combine is still the default. The user calls
> > > .setCombineHint(CombineHint) to make a selection (I think this was
> > > originally overloaded on DataSet and it looks like the pr1517
> > documentation
> > > update does not reflect the new usage).
> > >
> > > I'd be glad to merge this in but I didn't want to preempt Gabor or
> > Fabian.
> > >
> > > Greg
> > >
> > > On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uce@apache.org> wrote:
> > >
> > > > Thanks for the feedback. I would like to create a preview RC0 this
> > > > week like we did for the last releases, too. In past major releases,
> > > > we needed to create many release candidates, often for fixing just
> > > > some small issues. I would like to speed up the release process by
> > > > collecting as many issues as possible now with the RC0. Once these
> > > > issues are resolved, we can start voting with the RC1. This
> > > > essentially means that we have a feature freeze. I would create the
> > > > release-1.1 branch with RC1.
> > > >
> > > > Regarding the open issues:
> > > >
> > > > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > > > headers (#2194) and the savepoint disposal PR needs addressing of
> > > > comments.
> > > >
> > > > - The Cassandra connector fixes are either merged or very close to be
> > > > merged.
> > > >
> > > > - Metrics docs are not a blocker since the online docs are updated
> > > > after the release. Regarding the renaming, we should decide soon. Any
> > > > opinions here?
> > > >
> > > > - The YARN issues have been resolved.
> > > >
> > > > I will also make a pass over JIRA and our PRs to check whether we've
> > > > missed something.
> > > >
> > > > @Greg: You are right, the hash-based combine PR has been extensively
> > > > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > > > look into it and cannot tell whether we should merge it now. Is the
> > > > hash-based combine strategy activated by default or does the user has
> > > > to activate it manually? The later case would make me feel more
> > > > comfortable merging it for the release.
> > > >
> > > >
> > > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mxm@apache.org>
> > > wrote:
> > > > > Yet another one for the release: FLINK-4144
> > > > > https://github.com/apache/flink/pull/2191
> > > > >
> > > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mxm@apache.org
> >
> > > > wrote:
> > > > >> There is also FLINK-4141. We just found this during testing.
PR is
> > > > >> waiting to be merged here:
> > https://github.com/apache/flink/pull/2190
> > > > >>
> > > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <
> mxm@apache.org>
> > > > wrote:
> > > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this
> > release
> > > > >>> and actually a bit tricky to fix. I've addressed the other
issues
> > and
> > > > >>> merged all pending pull requests. Good to go from my side.
> > > > >>>
> > > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> > > chesnay@apache.org>
> > > > wrote:
> > > > >>>> There are also 2 fixes for Cassandra that  should be
included:
> > > > >>>> https://github.com/apache/flink/pull/2167
> > > > >>>> https://github.com/apache/flink/pull/2183
> > > > >>>>
> > > > >>>> We should also include the documentation for the metrics
stuff
> > > > (hopefully
> > > > >>>> merged today)
> > > > >>>> https://github.com/apache/flink/pull/2158
> > > > >>>>
> > > > >>>> In regards to metrics: To add a counter metric a user
currently
> > has
> > > > to call
> > > > >>>> "counter(...)" on
> > > > >>>> a MetricGroup. The point was raised in the documentation
PR that
> > we
> > > > may want
> > > > >>>> to give
> > > > >>>> them a more descriptive name like "addCounter(...)".
> > > > >>>>
> > > > >>>> I would be in favor of changing them but would like others
to
> > weigh
> > > > in on
> > > > >>>> this. IMO we
> > > > >>>> should nail this down before 1.1 .
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>> Chesnay
> > > > >>>>
> > > > >>>>
> > > > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > > > >>>>>
> > > > >>>>> It would be great if hash-based combine (FLINK-3477)
could make
> > it
> > > > in to
> > > > >>>>> be
> > > > >>>>> tested for this release. We've seen impressive improvements
in
> > > > performance
> > > > >>>>> (though, admittedly, some sort-based enhancements
are yet to be
> > > > worked
> > > > >>>>> on).
> > > > >>>>> This PR looks to be ripe.
> > > > >>>>>
> > > > >>>>> Also, as we tidy up a few things with Gelly and documentation,
> > what
> > > > is the
> > > > >>>>> schedule for a feature freeze and creating a 1.1
branch off
> > master?
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>> Greg
> > > > >>>>>
> > > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> > > rmetzger@apache.org
> > > > >
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the
release
> > management.
> > > > >>>>>> I'll be on vacation in three weeks, for three
weeks and I'm
> not
> > > > sure if
> > > > >>>>>> we
> > > > >>>>>> get the release done until then.
> > > > >>>>>>
> > > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi
<uce@apache.org
> >
> > > > wrote:
> > > > >>>>>>
> > > > >>>>>>> I would like to do it if that's OK with you
Robert. I would
> > > follow
> > > > >>>>>>> your suggestion and wait a few days until
the following
> > important
> > > > >>>>>>> fixes are in:
> > > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067
and
> > > > >>>>>>> https://github.com/apache/flink/pull/2083)
> > > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > > > >>>>>>> - Table API time support (
> > > > https://github.com/apache/flink/pull/2150)
> > > > >>>>>>> - Kafka at-least-once Producer (
> > > > >>>>>>
> > > > >>>>>> https://github.com/apache/flink/pull/2108)
> > > > >>>>>>>
> > > > >>>>>>> - Cassandra connector fixes (
> > > > https://github.com/apache/flink/pull/2163)
> > > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904
@Max: is there
> > > > something
> > > > >>>>>>
> > > > >>>>>> else?)
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger
<
> > > > rmetzger@apache.org>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> Hi,
> > > > >>>>>>>> it doesn't seem that there are volunteers
for the RM, so
> I'll
> > > > probably
> > > > >>>>>>
> > > > >>>>>> do
> > > > >>>>>>>>
> > > > >>>>>>>> it.
> > > > >>>>>>>>
> > > > >>>>>>>> I try to do the first release candidate
(mostly for testing)
> > > next
> > > > week
> > > > >>>>>>>
> > > > >>>>>>> (it
> > > > >>>>>>>>
> > > > >>>>>>>> depends on the JIRAs fixed by then)
> > > > >>>>>>>>
> > > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry
Saputra <
> > > > >>>>>>
> > > > >>>>>> henry.saputra@gmail.com
> > > > >>>>>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Thanks for the reply, @Max. I was
not aware it was about
> > > dynamic
> > > > >>>>>>>
> > > > >>>>>>> scaling,
> > > > >>>>>>>>>
> > > > >>>>>>>>> which I think also asked for YARN
support.
> > > > >>>>>>>>> I agree to list all related half
merge JIRA for the
> > > > ResourceManager.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Looking forward for the Apache Mesos
integration design for
> > > sure
> > > > =)
> > > > >>>>>>>>>
> > > > >>>>>>>>> - Henry
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM,
Maximilian Michels <
> > > > mxm@apache.org>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hi Robert, hi Henry,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> +1 for a 1.1.0 release soon!
We have enough new features
> > that
> > > > >>>>>>
> > > > >>>>>> justify
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> a major release.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> @Henry We have plans to extend
the ResourceManager to
> > interact
> > > > with
> > > > >>>>>>>>>> the Scheduler which will be a
prerequisite for dynamic
> > > scaling.
> > > > I
> > > > >>>>>>>>>> think this is out of scope for
1.1.0. The upcoming Mesos
> > > > integration
> > > > >>>>>>>>>> won't require additional refactoring
of the
> ResourceManager.
> > > > >>>>>>
> > > > >>>>>> Instead,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> we will create a new "Dispatcher"
component that takes
> care
> > of
> > > > >>>>>>>>>> bootstrapping the initial node
with the
> > > > JobManager/ResourceManager.
> > > > >>>>>>>>>>  From there on, everything will
be handled by the Mesos
> > > > >>>>>>>>>> ResourceManager. I recently discussed
this with Eron (CC)
> > who
> > > > came
> > > > >>>>>>
> > > > >>>>>> up
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> with this design and he plans
to publish it to the mailing
> > > list
> > > > >>>>>>
> > > > >>>>>> soon.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> How about listing relevant JIRA
issues here? "Half Merged"
> > is
> > > > kind
> > > > >>>>>>
> > > > >>>>>> of
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> hard to get for people who are
not involved in the
> different
> > > > >>>>>>>>>> components.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> The Cassandra adapter seems like
a pretty important thing
> to
> > > > have
> > > > >>>>>>
> > > > >>>>>> for
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> the next release. In addition,
I would like to merge
> > > FLINK-3667
> > > > and
> > > > >>>>>>>>>> FLINK-3937. Robert is doing a
review at the moment :)
> Those
> > > are
> > > > a)
> > > > >>>>>>>>>> refactoring of the command-line
and client classes b)
> adding
> > > > >>>>>>>>>> capability to resume cluster
programmatically.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Then we should also have a look
at any other
> critical/major
> > > bugs
> > > > >>>>>>>
> > > > >>>>>>> listed
> > > > >>>>>>>>>
> > > > >>>>>>>>> in
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> JIRA.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Cheers,
> > > > >>>>>>>>>> Max
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50
PM, Henry Saputra <
> > > > >>>>>>>
> > > > >>>>>>> henry.saputra@gmail.com
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Hi Robert,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks for staying the discussion.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Do you know if there any
open tasks for the Resource
> > Manager
> > > > left?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> That is probably needed for
Mesos integration?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> - Henry
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55
PM, Robert Metzger <
> > > > >>>>>>>
> > > > >>>>>>> rmetzger@apache.org>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Flink 1.0.0 was released
early March, so three months
> have
> > > > passed
> > > > >>>>>>>
> > > > >>>>>>> and
> > > > >>>>>>>>>
> > > > >>>>>>>>> I
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> think we should start
discussing the scope of the next
> > major
> > > > >>>>>>>
> > > > >>>>>>> release
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> (1.1.0).
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>  From a high level point
of view, we've added the
> > following
> > > > new
> > > > >>>>>>>>>
> > > > >>>>>>>>> features:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> in master:
> > > > >>>>>>>>>>>> - Table API Refactoring,
SQL, StreamSQL
> > > > >>>>>>>>>>>> - The metrics system
> > > > >>>>>>>>>>>> - Kinesis Connector
> > > > >>>>>>>>>>>> - Persistent file sources
for streaming
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Half merged:
> > > > >>>>>>>>>>>> - Resource manager refactoring
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Unmerged features:
> > > > >>>>>>>>>>>> - Cassandra connector
> > > > >>>>>>>>>>>> - Key groups ("rescaling
from savepoints")
> > > > >>>>>>>>>>>> - Queryable state
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I'm pretty sure I forgot
many other features / pull
> > > requests,
> > > > >>>>>>>
> > > > >>>>>>> please
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> post
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> them to this thread.
I'll collect them and create a Wiki
> > > page
> > > > out
> > > > >>>>>>>
> > > > >>>>>>> of
> > > > >>>>>>>>>
> > > > >>>>>>>>> it.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Some immediate TODOs
for us:
> > > > >>>>>>>>>>>> - Which of the unmerged
features are we going to add to
> > the
> > > > >>>>>>>
> > > > >>>>>>> release?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> - Which blockers do we
need to address before releasing?
> > > > >>>>>>>>>>>> - Are there any volunteers
for the release manager?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Regards,
> > > > >>>>>>>>>>>> Robert
> > > > >>>>>>>>>>>>
> > > > >>>>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message