flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: [DISCUSS] Releasing Flink 1.1.0
Date Tue, 05 Jul 2016 15:53:35 GMT
I think it's not strictly required that all code be in Scala. There is
already some Java code in there so we shouldn't force people to write Scala
code if they make a valuable contribution in Java.

On Tue, 5 Jul 2016 at 17:33 Flavio Pompermaier <pompermaier@okkam.it> wrote:

> Hi to all,
> if Flink 1.1 will introduce ufficially the Table API, do you think someone
> could take care of rewriting in scala the necessary java code of my PR
> about reading CSV as Rows instead of tuples[1]?
>
> For our use cases, and many new users approaching to Flink IMHO, that will
> be definitely useful because you can read csv files with more than 25
> columns and properly handle null values. What  do you think?
>
> [1] https://github.com/apache/flink/pull/1989
>
> Best,
> Flavio
> On 5 Jul 2016 18:08, "Till Rohrmann" <trohrmann@apache.org> wrote:
>
> I found another critical issue [1]. The murmur hash correction introduced
> between Flink 1.0 and 1.1 breaks the backwards compatibility with respect
> to savepoints. I think we have to fix this for the release.
>
> @Ufuk, I'm not sure whether I find time this week to work on FLINK-4150. I
> could make it a stretch goal for this sprint. Or we schedule it in next
> week's sprint.
>
> [1] https://issues.apache.org/jira/browse/FLINK-4154
>
>
>
> On Tue, Jul 5, 2016 at 3:56 PM, Greg Hogan <code@greghogan.com> wrote:
>
> > Hi Ufuk,
> >
> > The old sort-based combine is still the default. The user calls
> > .setCombineHint(CombineHint) to make a selection (I think this was
> > originally overloaded on DataSet and it looks like the pr1517
> documentation
> > update does not reflect the new usage).
> >
> > I'd be glad to merge this in but I didn't want to preempt Gabor or
> Fabian.
> >
> > Greg
> >
> > On Mon, Jul 4, 2016 at 7:47 AM, Ufuk Celebi <uce@apache.org> wrote:
> >
> > > Thanks for the feedback. I would like to create a preview RC0 this
> > > week like we did for the last releases, too. In past major releases,
> > > we needed to create many release candidates, often for fixing just
> > > some small issues. I would like to speed up the release process by
> > > collecting as many issues as possible now with the RC0. Once these
> > > issues are resolved, we can start voting with the RC1. This
> > > essentially means that we have a feature freeze. I would create the
> > > release-1.1 branch with RC1.
> > >
> > > Regarding the open issues:
> > >
> > > - The savepoint fixes are not yet in. There is a PR for the savepoint
> > > headers (#2194) and the savepoint disposal PR needs addressing of
> > > comments.
> > >
> > > - The Cassandra connector fixes are either merged or very close to be
> > > merged.
> > >
> > > - Metrics docs are not a blocker since the online docs are updated
> > > after the release. Regarding the renaming, we should decide soon. Any
> > > opinions here?
> > >
> > > - The YARN issues have been resolved.
> > >
> > > I will also make a pass over JIRA and our PRs to check whether we've
> > > missed something.
> > >
> > > @Greg: You are right, the hash-based combine PR has been extensively
> > > reviewed. Unfortunately, I'm not familiar with the code as I didn't
> > > look into it and cannot tell whether we should merge it now. Is the
> > > hash-based combine strategy activated by default or does the user has
> > > to activate it manually? The later case would make me feel more
> > > comfortable merging it for the release.
> > >
> > >
> > > On Fri, Jul 1, 2016 at 7:14 PM, Maximilian Michels <mxm@apache.org>
> > wrote:
> > > > Yet another one for the release: FLINK-4144
> > > > https://github.com/apache/flink/pull/2191
> > > >
> > > > On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels <mxm@apache.org>
> > > wrote:
> > > >> There is also FLINK-4141. We just found this during testing. PR is
> > > >> waiting to be merged here:
> https://github.com/apache/flink/pull/2190
> > > >>
> > > >> On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels <mxm@apache.org>
> > > wrote:
> > > >>> FLINK-3904 is not Yarn related. Not pressing to fix for this
> release
> > > >>> and actually a bit tricky to fix. I've addressed the other issues
> and
> > > >>> merged all pending pull requests. Good to go from my side.
> > > >>>
> > > >>> On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler <
> > chesnay@apache.org>
> > > wrote:
> > > >>>> There are also 2 fixes for Cassandra that  should be included:
> > > >>>> https://github.com/apache/flink/pull/2167
> > > >>>> https://github.com/apache/flink/pull/2183
> > > >>>>
> > > >>>> We should also include the documentation for the metrics stuff
> > > (hopefully
> > > >>>> merged today)
> > > >>>> https://github.com/apache/flink/pull/2158
> > > >>>>
> > > >>>> In regards to metrics: To add a counter metric a user currently
> has
> > > to call
> > > >>>> "counter(...)" on
> > > >>>> a MetricGroup. The point was raised in the documentation PR
that
> we
> > > may want
> > > >>>> to give
> > > >>>> them a more descriptive name like "addCounter(...)".
> > > >>>>
> > > >>>> I would be in favor of changing them but would like others
to
> weigh
> > > in on
> > > >>>> this. IMO we
> > > >>>> should nail this down before 1.1 .
> > > >>>>
> > > >>>> Regards,
> > > >>>> Chesnay
> > > >>>>
> > > >>>>
> > > >>>> On 30.06.2016 22:59, Greg Hogan wrote:
> > > >>>>>
> > > >>>>> It would be great if hash-based combine (FLINK-3477) could
make
> it
> > > in to
> > > >>>>> be
> > > >>>>> tested for this release. We've seen impressive improvements
in
> > > performance
> > > >>>>> (though, admittedly, some sort-based enhancements are
yet to be
> > > worked
> > > >>>>> on).
> > > >>>>> This PR looks to be ripe.
> > > >>>>>
> > > >>>>> Also, as we tidy up a few things with Gelly and documentation,
> what
> > > is the
> > > >>>>> schedule for a feature freeze and creating a 1.1 branch
off
> master?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Greg
> > > >>>>>
> > > >>>>> On Mon, Jun 27, 2016 at 7:23 AM, Robert Metzger <
> > rmetzger@apache.org
> > > >
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Sure Ufuk! Thanks a lot for taking care of the release
> management.
> > > >>>>>> I'll be on vacation in three weeks, for three weeks
and I'm not
> > > sure if
> > > >>>>>> we
> > > >>>>>> get the release done until then.
> > > >>>>>>
> > > >>>>>> On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi <uce@apache.org>
> > > wrote:
> > > >>>>>>
> > > >>>>>>> I would like to do it if that's OK with you Robert.
I would
> > follow
> > > >>>>>>> your suggestion and wait a few days until the
following
> important
> > > >>>>>>> fixes are in:
> > > >>>>>>> - Savepoint headers and proper disposal (FLINK-4067
and
> > > >>>>>>> https://github.com/apache/flink/pull/2083)
> > > >>>>>>> - Metrics (https://github.com/apache/flink/pull/2146)
> > > >>>>>>> - Table API time support (
> > > https://github.com/apache/flink/pull/2150)
> > > >>>>>>> - Kafka at-least-once Producer (
> > > >>>>>>
> > > >>>>>> https://github.com/apache/flink/pull/2108)
> > > >>>>>>>
> > > >>>>>>> - Cassandra connector fixes (
> > > https://github.com/apache/flink/pull/2163)
> > > >>>>>>> - YARN client fixes (FLINK-3675, FLINK-3904 @Max:
is there
> > > something
> > > >>>>>>
> > > >>>>>> else?)
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Thu, Jun 23, 2016 at 1:33 PM, Robert Metzger
<
> > > rmetzger@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Hi,
> > > >>>>>>>> it doesn't seem that there are volunteers
for the RM, so I'll
> > > probably
> > > >>>>>>
> > > >>>>>> do
> > > >>>>>>>>
> > > >>>>>>>> it.
> > > >>>>>>>>
> > > >>>>>>>> I try to do the first release candidate (mostly
for testing)
> > next
> > > week
> > > >>>>>>>
> > > >>>>>>> (it
> > > >>>>>>>>
> > > >>>>>>>> depends on the JIRAs fixed by then)
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra
<
> > > >>>>>>
> > > >>>>>> henry.saputra@gmail.com
> > > >>>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thanks for the reply, @Max. I was not
aware it was about
> > dynamic
> > > >>>>>>>
> > > >>>>>>> scaling,
> > > >>>>>>>>>
> > > >>>>>>>>> which I think also asked for YARN support.
> > > >>>>>>>>> I agree to list all related half merge
JIRA for the
> > > ResourceManager.
> > > >>>>>>>>>
> > > >>>>>>>>> Looking forward for the Apache Mesos integration
design for
> > sure
> > > =)
> > > >>>>>>>>>
> > > >>>>>>>>> - Henry
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jun 16, 2016 at 2:12 AM, Maximilian
Michels <
> > > mxm@apache.org>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Robert, hi Henry,
> > > >>>>>>>>>>
> > > >>>>>>>>>> +1 for a 1.1.0 release soon! We have
enough new features
> that
> > > >>>>>>
> > > >>>>>> justify
> > > >>>>>>>>>>
> > > >>>>>>>>>> a major release.
> > > >>>>>>>>>>
> > > >>>>>>>>>> @Henry We have plans to extend the
ResourceManager to
> interact
> > > with
> > > >>>>>>>>>> the Scheduler which will be a prerequisite
for dynamic
> > scaling.
> > > I
> > > >>>>>>>>>> think this is out of scope for 1.1.0.
The upcoming Mesos
> > > integration
> > > >>>>>>>>>> won't require additional refactoring
of the ResourceManager.
> > > >>>>>>
> > > >>>>>> Instead,
> > > >>>>>>>>>>
> > > >>>>>>>>>> we will create a new "Dispatcher"
component that takes care
> of
> > > >>>>>>>>>> bootstrapping the initial node with
the
> > > JobManager/ResourceManager.
> > > >>>>>>>>>>  From there on, everything will be
handled by the Mesos
> > > >>>>>>>>>> ResourceManager. I recently discussed
this with Eron (CC)
> who
> > > came
> > > >>>>>>
> > > >>>>>> up
> > > >>>>>>>>>>
> > > >>>>>>>>>> with this design and he plans to publish
it to the mailing
> > list
> > > >>>>>>
> > > >>>>>> soon.
> > > >>>>>>>>>>
> > > >>>>>>>>>> How about listing relevant JIRA issues
here? "Half Merged"
> is
> > > kind
> > > >>>>>>
> > > >>>>>> of
> > > >>>>>>>>>>
> > > >>>>>>>>>> hard to get for people who are not
involved in the different
> > > >>>>>>>>>> components.
> > > >>>>>>>>>>
> > > >>>>>>>>>> The Cassandra adapter seems like a
pretty important thing to
> > > have
> > > >>>>>>
> > > >>>>>> for
> > > >>>>>>>>>>
> > > >>>>>>>>>> the next release. In addition, I would
like to merge
> > FLINK-3667
> > > and
> > > >>>>>>>>>> FLINK-3937. Robert is doing a review
at the moment :) Those
> > are
> > > a)
> > > >>>>>>>>>> refactoring of the command-line and
client classes b) adding
> > > >>>>>>>>>> capability to resume cluster programmatically.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Then we should also have a look at
any other critical/major
> > bugs
> > > >>>>>>>
> > > >>>>>>> listed
> > > >>>>>>>>>
> > > >>>>>>>>> in
> > > >>>>>>>>>>
> > > >>>>>>>>>> JIRA.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Cheers,
> > > >>>>>>>>>> Max
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, Jun 15, 2016 at 10:50 PM,
Henry Saputra <
> > > >>>>>>>
> > > >>>>>>> henry.saputra@gmail.com
> > > >>>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hi Robert,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for staying the discussion.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Do you know if there any open
tasks for the Resource
> Manager
> > > left?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That is probably needed for Mesos
integration?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Henry
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Jun 15, 2016 at 12:55
PM, Robert Metzger <
> > > >>>>>>>
> > > >>>>>>> rmetzger@apache.org>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Flink 1.0.0 was released early
March, so three months have
> > > passed
> > > >>>>>>>
> > > >>>>>>> and
> > > >>>>>>>>>
> > > >>>>>>>>> I
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> think we should start discussing
the scope of the next
> major
> > > >>>>>>>
> > > >>>>>>> release
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> (1.1.0).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>  From a high level point of
view, we've added the
> following
> > > new
> > > >>>>>>>>>
> > > >>>>>>>>> features:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> in master:
> > > >>>>>>>>>>>> - Table API Refactoring, SQL,
StreamSQL
> > > >>>>>>>>>>>> - The metrics system
> > > >>>>>>>>>>>> - Kinesis Connector
> > > >>>>>>>>>>>> - Persistent file sources
for streaming
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Half merged:
> > > >>>>>>>>>>>> - Resource manager refactoring
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Unmerged features:
> > > >>>>>>>>>>>> - Cassandra connector
> > > >>>>>>>>>>>> - Key groups ("rescaling from
savepoints")
> > > >>>>>>>>>>>> - Queryable state
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I'm pretty sure I forgot many
other features / pull
> > requests,
> > > >>>>>>>
> > > >>>>>>> please
> > > >>>>>>>>>>
> > > >>>>>>>>>> post
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> them to this thread. I'll
collect them and create a Wiki
> > page
> > > out
> > > >>>>>>>
> > > >>>>>>> of
> > > >>>>>>>>>
> > > >>>>>>>>> it.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Some immediate TODOs for us:
> > > >>>>>>>>>>>> - Which of the unmerged features
are we going to add to
> the
> > > >>>>>>>
> > > >>>>>>> release?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - Which blockers do we need
to address before releasing?
> > > >>>>>>>>>>>> - Are there any volunteers
for the release manager?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>> Robert
> > > >>>>>>>>>>>>
> > > >>>>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message