flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Planning Flink release 0.7-incubating
Date Tue, 07 Oct 2014 20:40:22 GMT
I would add #142.

On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <sewen@apache.org> wrote:

> I agree, having a release candidate out would be nice.
>
> What is your opinion on issue #142? Fault tolerance is inactive by default,
> but can be activated through the configuration.
>
> On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
> > Oh. I didn't know this. The last mail from Gyula sounded like the
> Streaming
> > part is ready2release.
> >
> > My personal goal was to prepare a release candiate today so that we have
> a
> > reference point against which we can test (which would also mean forking
> of
> > a 0.7-release branch and basically doing a feature freeze).
> > I also though that it might be a good idea to have a release candidate by
> > tomorrow because it could be of use for the Hackathon tomorrow in
> > Stockholm.
> > But since the Scala POJO changes and the streaming examples are both
> > outstanding, we can probably scratch that.
> >
> > Doing a fork, release candidate and a feature freeze doesn't mean we can
> > not supply bugfixes anymore for the 0.7-release ;)
> >
> > What is your best-case scenario for FLINK-1103? (as far as I can see the
> > only outstanding streaming feature for the release)
> >
> > Best,
> > Robert
> >
> >
> > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <balassi.marton@gmail.com
> >
> > wrote:
> >
> > > As for the streaming side we would like to push some bugfix commits and
> > the
> > > resolution of the FLINK-1103 JIRA issue.
> > >
> > > These are more or less ready, hopefully will be available at the end of
> > > this week.
> > >
> > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <rmetzger@apache.org>
> > > wrote:
> > >
> > > > I've merged the record api deprecation already.
> > > > I'll merge #141 once Aljoscha provided his Scala changes.
> > > >
> > > > We certainly should merge #136 and #143 as well.
> > > >
> > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <fhueske@apache.org>
> > > wrote:
> > > >
> > > > > So which PRs will be included in the candidate?
> > > > >
> > > > > #141 POJOs
> > > > > #144 Deprecate Record API
> > > > > #136 Fixed example quickstart
> > > > > #143(?) Hadoop Compat: Documentation + Hadoop function wrappers
> > > (includes
> > > > > PR #131)
> > > > >
> > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <sewen@apache.org>:
> > > > >
> > > > > > Great news, looking forward to seeing this in the master!
> > > > > >
> > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <
> > rmetzger@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > As an update for everyone: My POJO feature is finished,
> including
> > > > > > > documentation.
> > > > > > > Aljoscha is currently adopting the Scala API to have support
> for
> > > > > (nested)
> > > > > > > POJOs as well.
> > > > > > > Once that is done, I'll merge everything and create a first
> > > candidate
> > > > > > that
> > > > > > > we can use for testing.
> > > > > > >
> > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske <
> > > fhueske@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I just checked the "Run Example" quickstart and it
needs a
> bit
> > of
> > > > > work.
> > > > > > > >
> > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <
> rmetzger@apache.org
> > >:
> > > > > > > >
> > > > > > > > > I'm working hard on getting the POJOs ready.
> > > > > > > > >
> > > > > > > > > We also should do a pass over our documentation,
the
> > > quickstarts
> > > > > and
> > > > > > > the
> > > > > > > > > website to see if everything is in a good state
(for
> example
> > > the
> > > > > > > > > collection-based execution needs documentation
as well). We
> > > > should
> > > > > > also
> > > > > > > > > finally document the hadoop-input format wrappers
(I think
> > Timo
> > > > is
> > > > > > > > working
> > > > > > > > > on a pull request for that).
> > > > > > > > > This page mentions the LocalDistributedExecutor
and
> contains
> > > some
> > > > > > (most
> > > > > > > > > likely outdated) scala code:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
> > > > > > > > >
> > > > > > > > > We also need to deprecate the old record api
(
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106).
> > > > > > > > >
> > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen
<
> > > sewen@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I think we are approaching ready state.
> > > > > > > > > > Last issues are going in and we started
working on
> > > dependencies
> > > > > and
> > > > > > > > test
> > > > > > > > > > platform diversity in order to make stabilizing
phase.
> > > > > > > > > >
> > > > > > > > > > We should have an official feature freeze
soon and fork
> the
> > > > > > > 0.7-release
> > > > > > > > > > branch. I personally vote to include the
POJO support (I
> > > think
> > > > > > Robert
> > > > > > > > is
> > > > > > > > > > sorting that one out and is close to completion),
and I
> > want
> > > to
> > > > > add
> > > > > > > the
> > > > > > > > > > collection-based execution (today or tomorrow).
Till has
> > the
> > > > BLOB
> > > > > > > > manager
> > > > > > > > > > ready, which would be good to include (better
support
> large
> > > > > > libraries
> > > > > > > > or
> > > > > > > > > > fat jars).
> > > > > > > > > >
> > > > > > > > > > After that, I vote to freeze.
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora
<
> > > gyfora@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey,
> > > > > > > > > > >
> > > > > > > > > > > So what is the current decision regarding
the time of
> the
> > > > > > upcoming
> > > > > > > > > > release?
> > > > > > > > > > >
> > > > > > > > > > > As for the streaming component, we
included all the
> > > features
> > > > we
> > > > > > > > wanted,
> > > > > > > > > > we
> > > > > > > > > > > will start to test everything tomorrow,
making sure
> that
> > > all
> > > > > > works
> > > > > > > as
> > > > > > > > > > > intended.
> > > > > > > > > > > We are also almost finished with cleaning
up the
> > connector
> > > > > > > > dependencies
> > > > > > > > > > > that Stephan pointed out, should be
finished by
> tomorrow.
> > > > > > > > > > >
> > > > > > > > > > > Gyula
> > > > > > > > > > >
> > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske
<
> > > fhueske@apache.org>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1 to manage this on JIRA (if
possible)
> > > > > > > > > > > >
> > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha
Krettek <
> > > > > > aljoscha@apache.org
> > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > >> Can we not manage this stuff
on Jira?
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16
AM, Stephan Ewen <
> > > > > > > sewen@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >>> I personally like the
idea of SOFT time-based
> feature
> > > > > > freezes.
> > > > > > > > > > > Otherwise,
> > > > > > > > > > > >>> releases will get delayed
again and again, because
> of
> > > > > > features
> > > > > > > > that
> > > > > > > > > > we
> > > > > > > > > > > >> try
> > > > > > > > > > > >>> to squeeze in.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Having reached the freeze
point already, we could
> > still
> > > > > > include
> > > > > > > > the
> > > > > > > > > > > >>> features that are pending
ready state in the next
> > days
> > > > > > > > (streaming,
> > > > > > > > > > blob
> > > > > > > > > > > >>> Manager, POJOs), but otherwise
head for a release
> > > state.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> We had a mail listing
the issues to include into
> 0.7,
> > > > but a
> > > > > > > wiki
> > > > > > > > > page
> > > > > > > > > > > >> would
> > > > > > > > > > > >>> probably be better. In
that sense, we could start
> > > > > collecting
> > > > > > > the
> > > > > > > > > > issues
> > > > > > > > > > > >> for
> > > > > > > > > > > >>> the next release from
now on.
> > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb
"Daniel Warneke" <
> > > > > > > warneke@apache.org
> > > > > > > > >:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> Hi,
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> I like Fabian's idea.
Is there a wiki page (or
> > > something
> > > > > > > > similar)
> > > > > > > > > > > where
> > > > > > > > > > > >> we
> > > > > > > > > > > >>>> can collect the proposed
JIRAs?
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Best regards,
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>    Daniel
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Am 24.09.2014 23:03,
schrieb Fabian Hueske:
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> I agree, a hard
feature stop deadline might not
> be
> > > the
> > > > > best
> > > > > > > > > > practice.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> How about the
following procedure:
> > > > > > > > > > > >>>>> We decide two
(or three) weeks before a targeted
> > > > release
> > > > > > date
> > > > > > > > > about
> > > > > > > > > > > >> which
> > > > > > > > > > > >>>>> JIRAs to include.
JIRAs that are selected for a
> > > release
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > >>>>> completed
> > > > > > > > > > > >>>>> or really close
to completion (via progress
> > estimates
> > > > in
> > > > > > > JIRA).
> > > > > > > > > > > >>>>> After we decided
which JIRAs to include in a
> > release,
> > > > we
> > > > > > can
> > > > > > > > use
> > > > > > > > > > JIRA
> > > > > > > > > > > >> to
> > > > > > > > > > > >>>>> track the progress
and dedicate another week
> > > > exclusively
> > > > > > for
> > > > > > > > > > testing
> > > > > > > > > > > >> after
> > > > > > > > > > > >>>>> the last feature
was completed.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> 2014-09-24 19:10
GMT+02:00 Márton Balassi <
> > > > > > > > > > balassi.marton@gmail.com
> > > > > > > > > > > >:
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> As for the streaming
team we're also getting
> ready
> > > for
> > > > > the
> > > > > > > > > release,
> > > > > > > > > > > >> but a
> > > > > > > > > > > >>>>>> couple of
days will be needed to finish the
> > features
> > > > > that
> > > > > > we
> > > > > > > > > would
> > > > > > > > > > > >> like
> > > > > > > > > > > >>>>>> to
> > > > > > > > > > > >>>>>> include.
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>    - A little
work is still needed for reduce
> > > > operations
> > > > > > and
> > > > > > > > > > > >>>>>>    groups/connected
streams (any comment on
> > Gyula's
> > > > > recent
> > > > > > > > > e-mail
> > > > > > > > > > is
> > > > > > > > > > > >>>>>> really
> > > > > > > > > > > >>>>>>    appreciated
:) )
> > > > > > > > > > > >>>>>>    - The examples
are being updated to match the
> > > > > standard,
> > > > > > > > check
> > > > > > > > > > out
> > > > > > > > > > > >> the
> > > > > > > > > > > >>>>>>    WordCount.
(
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>
> https://github.com/mbalassi/incubator-flink/blob/
> > > > > > > > > > > >>>>>>
> streaming-new/flink-addons/flink-streaming/flink-
> > > > > > > > > > > >>>>>>
> streaming-examples/src/main/java/org/apache/flink/
> > > > > > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java
> > > > > > > > > > > >>>>>> )
> > > > > > > > > > > >>>>>>    Hopefully
it gives you some deja vu. :)
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> On Wed, Sep
24, 2014 at 6:53 PM, Ufuk Celebi <
> > > > > > > uce@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> On 24 Sep
2014, at 18:37, Robert Metzger <
> > > > > > > rmetzger@apache.org
> > > > > > > > >
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> Hey guys,
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> exactly
3 weeks ago, we discussed to do a
> > feature
> > > > > freeze
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > >>>>>>>> 0.7-incubating
release today.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> From
our initial feature list:
> > > > > > > > > > > >>>>>>>> -
*Flink Streaming* "Beta Preview". I would
> > > suggest
> > > > to
> > > > > > > ship
> > > > > > > > > the
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> streaming,
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> but
clearly mark it as a preview in the
> > > > documentation.
> > > > > > > > > > > >>>>>>>> -*
Java API Pojo improvements*: Code
> generation,
> > > key
> > > > > > > > selection
> > > > > > > > > > > >> using a
> > > > > > > > > > > >>>>>>>> string-expression:
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032
> > > > > > > > > > > >>>>>>>>  -
*Reworked Scala API*. Bring the Scala API
> in
> > > sync
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > >> latest
> > > > > > > > > > > >>>>>>>> developments
in the Java API:
> > > > > > > > > > > >>>>>>>>
> https://issues.apache.org/jira/browse/FLINK-641
> > > > > > > > > > > >>>>>>>>  -*
Akka-based RPC service*:
> > > > > > > > > > > >>>>>>>>
> > https://issues.apache.org/jira/browse/FLINK-1019
> > > > > > > > > > > >>>>>>>>  -
*Kryo-based serialization*. This feature
> has
> > > been
> > > > > > > > requested
> > > > > > > > > > by
> > > > > > > > > > > >> many
> > > > > > > > > > > >>>>>>>> users.
Mostly because they wanted to use
> > > Collections
> > > > > > > inside
> > > > > > > > > > POJOs:
> > > > > > > > > > > >>>>>>>>
> https://issues.apache.org/jira/browse/FLINK-610
> > > > > > > > > > > >>>>>>>> -
Rework JobManager internals to support
> > > incremental
> > > > > > > program
> > > > > > > > > > > >> rollout &
> > > > > > > > > > > >>>>>>>> execution
> > > > > > > > > > > >>>>>>>> -
First parts of dynamic memory assignments
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> The
following features are in the master, as
> of
> > > > today:
> > > > > > > > > > > >>>>>>>> -
*Flink Streaming*
> > > > > > > > > > > >>>>>>>> -
*Reworked Scala API*
> > > > > > > > > > > >>>>>>>> -*
New Scheduler*
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> We
certainly need some days to test everything
> > > until
> > > > > we
> > > > > > > can
> > > > > > > > > > start
> > > > > > > > > > > >> the
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> vote.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> Based
on our experience with the last major
> > > > release, I
> > > > > > > would
> > > > > > > > > > > really
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> like
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> to
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> do
the testing and bugfixing BEFORE the first
> > > > release
> > > > > > > > > candidate.
> > > > > > > > > > > For
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> 0.6-incubating
release, we had 6  candidates)
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> How
do you guys feel about this? Should we
> wait
> > a
> > > > few
> > > > > > more
> > > > > > > > > days
> > > > > > > > > > > for
> > > > > > > > > > > >> the
> > > > > > > > > > > >>>>>>>> release
so that a few more features make it
> into
> > > the
> > > > > > > > release?
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> I'm
undecided on this. On the one hand, its
> > really
> > > > > nice
> > > > > > to
> > > > > > > > > > release
> > > > > > > > > > > >> on a
> > > > > > > > > > > >>>>>>>> regular
schedule, but it also eats up some
> time
> > > and
> > > > > > causes
> > > > > > > > > > > overhead
> > > > > > > > > > > >>>>>>>> (different
branches etc.).
> > > > > > > > > > > >>>>>>>> I
would really like to have the Java API Pojo
> > > > > > improvements
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> release.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> I
think I can finish it until end of this
> week.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> Opinions?
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>> I agree
that the finished features (especially
> > the
> > > > > Scala
> > > > > > > API)
> > > > > > > > > are
> > > > > > > > > > > >> nice
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>> for
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> a new
release, but still I would like to wait a
> > few
> > > > > more
> > > > > > > > days.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> Some of
the missing features are on the brink
> of
> > > > being
> > > > > > > > finished
> > > > > > > > > > > (e.g.
> > > > > > > > > > > >>>>>>> the
> > > > > > > > > > > >>>>>>> Pojo improvements).
I wouldn't want to invest a
> > > week
> > > > in
> > > > > > bug
> > > > > > > > > > fixing
> > > > > > > > > > > >> and
> > > > > > > > > > > >>>>>>> doing
the release vote, when the new features
> are
> > > > > likely
> > > > > > to
> > > > > > > > be
> > > > > > > > > > > >> finished
> > > > > > > > > > > >>>>>>> just a
few days afterwards. And the upcoming
> > > features
> > > > > > will
> > > > > > > > > > > >> definitely be
> > > > > > > > > > > >>>>>>> worth
a release, so users can work with them.
> ;)
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message