apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <...@apache.org>
Subject Re: Malhar release 3.6
Date Wed, 07 Dec 2016 04:04:02 GMT
Thanks David.

Anyone else wants to contribute something to the release notes? It is time
to wrap it up.

Thanks,
Thomas


On Tue, Dec 6, 2016 at 3:14 PM, David Yan <david@datatorrent.com> wrote:

> For the work in the Windowed Operator that is included in the 3.6 release:
>
> There is now an option to use spillable data structures for the state
> storage of the Windowed Operator. This enables the Windowed Operator to
> store large states and perform efficient checkpointing.
>
> We also did benchmarking on WindowedOperator with the spillable data
> structures. From the result of our findings, we improved greatly how
> objects are serialized and reduced garbage collection considerably in the
> Managed State layer.
>
> Work is still in progress for purging state that is not needed any more and
> further improving the performance of Managed State that the spillable data
> structures depend on.
>
> Siyuan and Bright, please add to this if I'm missing anything.
>
> David
>
> On Fri, Dec 2, 2016 at 8:55 PM, Thomas Weise <thw@apache.org> wrote:
>
> > The release is available already. For the announcement, would anyone want
> > to provide some information that can go on the blog page, similar to what
> > we had last time:
> >
> > https://blogs.apache.org/apex/entry/apache_apex_malhar_3_5
> >
> > Thanks,
> > Thomas
> >
> >
> > On Tue, Nov 22, 2016 at 2:14 PM, Thomas Weise <thw@apache.org> wrote:
> >
> > > IMO we should not delay the release any longer. I will cut the RC
> > tomorrow
> > > unless there is an objection.
> > >
> > > Thomas
> > >
> > >
> > > On Tue, Nov 22, 2016 at 2:05 PM, David Yan <david@datatorrent.com>
> > wrote:
> > >
> > >> Thomas,
> > >>
> > >> We need more time for the timebucket change and the purging feature,
> and
> > >> we
> > >> need to do benchmarking after those features have been implemented.
> > >> Is it possible to postpone the release? If not, we can shoot for
> > >> completing
> > >> the features for the 3.7.0 release.
> > >>
> > >> David
> > >>
> > >> On Tue, Nov 22, 2016 at 12:08 PM, Thomas Weise <thw@apache.org>
> wrote:
> > >>
> > >> > David, Siyuan,
> > >> >
> > >> > Any update, I would like to cut the RC.
> > >> >
> > >> > Thanks
> > >> >
> > >> > On Sun, Nov 20, 2016 at 10:23 PM, Siyuan Hua <
> siyuan@datatorrent.com>
> > >> > wrote:
> > >> >
> > >> > > Yes yes,
> > >> > > We need to put the clear plan upfront
> > >> > >
> > >> > > This ticket was thought and scheduled to be done after the
> > >> > APEXMALHAR-2271
> > >> > > SpillableSetMultimap which bucketize elements on the time from
> > values.
> > >> > > But per discussion with David,  we realized this ticket is more
> > >> important
> > >> > > and affect the general purging functionality so I just jumped
to
> > this
> > >> > issue
> > >> > > late last week.
> > >> > > Now the status is the pull request is there. I feel it can be
> merged
> > >> > soon.
> > >> > > https://github.com/apache/apex-malhar/pull/503
> > >> > >
> > >> > > Regards,
> > >> > > Siyuan
> > >> > >
> > >> > > On Sat, Nov 19, 2016 at 12:56 AM, Thomas Weise <thw@apache.org>
> > >> wrote:
> > >> > >
> > >> > > > David,
> > >> > > >
> > >> > > > APEXMALHAR-2301 is still open. Are we on track to complete
the
> > >> JIRAs by
> > >> > > > Monday or should we move to next release?
> > >> > > >
> > >> > > > Siyuan,
> > >> > > >
> > >> > > > There is a question on the ticket from a month ago that
wasn't
> > >> > > addressed. I
> > >> > > > think there needs to be more visibility on the thought process
> and
> > >> > > > decisions made, otherwise it is very difficult for others
to
> > follow.
> > >> > Open
> > >> > > > collaboration is key.
> > >> > > >
> > >> > > > Thanks
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Nov 17, 2016 at 8:29 PM, David Yan <
> david@datatorrent.com
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > > Hi Thomas,
> > >> > > > >
> > >> > > > > We would like to finish the following tickets if possible
for
> > the
> > >> > 3.6.0
> > >> > > > > release:
> > >> > > > >
> > >> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2301
(ETA:
> > >> 11/18)
> > >> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2339
(ETA:
> > >> 11/21)
> > >> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2345
(ETA:
> > >> 11/21)
> > >> > > > >
> > >> > > > > For https://issues.apache.org/jira/browse/APEXMALHAR-2271,
> > since
> > >> > this
> > >> > > > only
> > >> > > > > affects Session Windows and still quite some work needs
to be
> > done
> > >> > > there,
> > >> > > > > we would like to defer that to the next release.
> > >> > > > >
> > >> > > > > Siyuan and Bright, please also chime in if I'm missing
> anything.
> > >> > > > >
> > >> > > > > David
> > >> > > > >
> > >> > > > > On Thu, Nov 17, 2016 at 2:49 AM, Thomas Weise <thw@apache.org
> >
> > >> > wrote:
> > >> > > > >
> > >> > > > > > David,
> > >> > > > > >
> > >> > > > > > Any update WRT APEXMALHAR-2130
> > >> > > > > > <https://issues.apache.org/jira/browse/APEXMALHAR-2130>
/
> > >> > > > > APEXMALHAR-2301
> > >> > > > > > <https://issues.apache.org/jira/browse/APEXMALHAR-2301>
?
> > >> > > > > >
> > >> > > > > > I would like to cut first RC by end of the week.
> > >> > > > > >
> > >> > > > > > Thanks
> > >> > > > > >
> > >> > > > > > On Wed, Nov 16, 2016 at 12:44 AM, Thomas Weise
<
> > thw@apache.org>
> > >> > > wrote:
> > >> > > > > >
> > >> > > > > > > It has been a while and the issues that we
were waiting
> for
> > >> are
> > >> > > still
> > >> > > > > > > unresolved.
> > >> > > > > > >
> > >> > > > > > > https://issues.apache.org/jira/issues/?jql=fixVersion%
> > >> > > > > > > 20%3D%203.6.0%20AND%20project%20%3D%20APEXMALHAR%20and%
> > >> > > > > > 20resolution%20%3D%
> > >> > > > > > > 20unresolved%20ORDER%20BY%20status%20ASC
> > >> > > > > > >
> > >> > > > > > > I would suggest:
> > >> > > > > > >
> > >> > > > > > > APEXMALHAR-2300 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2300
> > >> > > > > >
> > >> > > > > > -
> > >> > > > > > > defer
> > >> > > > > > > APEXMALHAR-2130 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2130
> > >> > > > > >
> > >> > > > > > /
> > >> > > > > > > APEXMALHAR-2301 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2301
> > >> > > > > >
> > >> > > > > > -
> > >> > > > > > > David/Siyuan can you please give an update
and
> > recommendation.
> > >> > > > > > > APEXMALHAR-2284 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2284
> > >> > > > > >
> > >> > > > > > -
> > >> > > > > > > see discussion on the PR and JIRA. Also discussed
it with
> > >> Bhupesh
> > >> > > and
> > >> > > > > > > Chinmay today and we think that this join
operator should
> > >> > probably
> > >> > > be
> > >> > > > > > > replaced with the new operator that is based
on
> > >> WindowOperator.
> > >> > > > > > > APEXMALHAR-2203 <https://issues.apache.org/
> > >> > > > jira/browse/APEXMALHAR-2203
> > >> > > > > >
> > >> > > > > > -
> > >> > > > > > > defer
> > >> > > > > > >
> > >> > > > > > > Please provide any feedback you may have
within a day so
> > that
> > >> we
> > >> > > can
> > >> > > > > > > unblock the release.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Thomas
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Mon, Nov 7, 2016 at 6:36 AM, Chaitanya
Chebolu <
> > >> > > > > > > chaitanya@datatorrent.com> wrote:
> > >> > > > > > >
> > >> > > > > > >> Hi Thomas,
> > >> > > > > > >>
> > >> > > > > > >>    I am working on APEXMALHAR-2284 and
will open a PR in
> > >> couple
> > >> > of
> > >> > > > > days.
> > >> > > > > > >>
> > >> > > > > > >> Regards,
> > >> > > > > > >> Chaitanya
> > >> > > > > > >>
> > >> > > > > > >> On Sun, Nov 6, 2016 at 10:51 PM, Thomas
Weise <
> > >> thw@apache.org>
> > >> > > > wrote:
> > >> > > > > > >>
> > >> > > > > > >> > Is anyone working on APEXMALHAR-2284
?
> > >> > > > > > >> >
> > >> > > > > > >> > On Fri, Oct 28, 2016 at 11:00 AM,
Bhupesh Chawda <
> > >> > > > > > >> bhupesh@datatorrent.com>
> > >> > > > > > >> > wrote:
> > >> > > > > > >> >
> > >> > > > > > >> > > +1
> > >> > > > > > >> > >
> > >> > > > > > >> > > ~ Bhupesh
> > >> > > > > > >> > >
> > >> > > > > > >> > > On Fri, Oct 28, 2016 at 2:29
PM, Tushar Gosavi <
> > >> > > > > > >> tushar@datatorrent.com>
> > >> > > > > > >> > > wrote:
> > >> > > > > > >> > >
> > >> > > > > > >> > > > +1
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > - Tushar.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > On Fri, Oct 28, 2016 at
12:52 PM, Aniruddha
> Thombare
> > >> > > > > > >> > > > <aniruddha@datatorrent.com>
wrote:
> > >> > > > > > >> > > > > +1
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Thanks,
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > A
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > _____________________________________
> > >> > > > > > >> > > > > Sent with difficulty,
I mean handheld ;)
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > On 28 Oct 2016 12:30
pm, "Siyuan Hua" <
> > >> > > > siyuan@datatorrent.com
> > >> > > > > >
> > >> > > > > > >> > wrote:
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > >> +1
> > >> > > > > > >> > > > >>
> > >> > > > > > >> > > > >> Thanks´╝ü
> > >> > > > > > >> > > > >>
> > >> > > > > > >> > > > >> Sent from my
iPhone
> > >> > > > > > >> > > > >>
> > >> > > > > > >> > > > >> > On Oct 26,
2016, at 13:11, Thomas Weise <
> > >> > > thw@apache.org>
> > >> > > > > > >> wrote:
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > Hi,
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > I'm proposing
another release of Malhar in
> > >> November.
> > >> > > > There
> > >> > > > > > are
> > >> > > > > > >> 49
> > >> > > > > > >> > > > issues
> > >> > > > > > >> > > > >> > marked for
the release, including important
> bug
> > >> > fixes,
> > >> > > > new
> > >> > > > > > >> > > > documentation,
> > >> > > > > > >> > > > >> > SQL support
and the work for windowed operator
> > >> state
> > >> > > > > > >> management:
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > https://issues.apache.org/
> > >> > jira/issues/?jql=fixVersion%
> > >> > > > > > >> > > > >> 20%3D%203.6.0%20AND%20project%
> > >> > > 20%3D%20APEXMALHAR%20ORDER%
> > >> > > > > > >> > > > >> 20BY%20status%20ASC
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > Currently
there is at least one blocker, the
> > join
> > >> > > > operator
> > >> > > > > is
> > >> > > > > > >> > broken
> > >> > > > > > >> > > > >> after
> > >> > > > > > >> > > > >> > change in
managed state. It also affects the
> SQL
> > >> > > feature.
> > >> > > > > > >> > > > >> >
> > >> > > > > > >> > > > >> > Thanks,
> > >> > > > > > >> > > > >> > Thomas
> > >> > > > > > >> > > > >>
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message