zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Olivelli <eolive...@gmail.com>
Subject Re: ZooKeeper 3.5 blocker issues
Date Thu, 20 Dec 2018 09:21:16 GMT
Il giorno gio 20 dic 2018 alle ore 10:20 Norbert Kalmar
<nkalmar@cloudera.com.invalid> ha scritto:
>
> One more thing - for the CI integration, actually, Java reports from Clover
> - by maven - should be already available on master. Basically we only
> analyse zookeeper-server, but as it is planned to be seperated, I wrote
> Clover so it starts from root pom and aggregates the results.

Got it.

>
> On Thu, Dec 20, 2018 at 10:17 AM Norbert Kalmar <nkalmar@cloudera.com>
> wrote:
>
> > Sure, sorry about that, I didn't check thoroughly the static analyser
> > part. Spotbugs works for me!
> >
> > Thanks Enrico!
> >
> > On Thu, Dec 20, 2018 at 10:10 AM Enrico Olivelli <eolivelli@gmail.com>
> > wrote:
> >
> >> Great
> >>
> >> Il giorno gio 20 dic 2018 alle ore 10:07 Norbert Kalmar
> >> <nkalmar@cloudera.com.invalid> ha scritto:
> >> >
> >> > Subtasks:
> >> > Findbugs, checkstyle -
> >> https://issues.apache.org/jira/browse/ZOOKEEPER-3223
> >>
> >> We don't have checksyle. In my experience introducing checkstyle break
> >> every pending patch.
> >> I would like to narrow down this issue to "Spotbugs" and pick in up
> >>
> >> > CI integration - https://issues.apache.org/jira/browse/ZOOKEEPER-3224
> >> I would like to pick this up
> >>
> >> Enrico
> >>
> >> > Code coverage - https://issues.apache.org/jira/browse/ZOOKEEPER-3225 -
> >> I
> >> > laready started this one and some of it is committed with the patch, so
> >> I
> >> > will continue to work on it.
> >> > Recipes and contrib -
> >> https://issues.apache.org/jira/browse/ZOOKEEPER-3171
> >> > - Already on it, recipes is done, PR soon available.
> >> > Assembly - https://issues.apache.org/jira/browse/ZOOKEEPER-3029
> >> >
> >> > This are the tasks left I can think of. If anything is missing, feel
> >> free
> >> > to create a jira, or let me know.
> >> > The ones I'm already working on - 3225, 3171 - I made a comment, Those
> >> > should be ready this week.
> >>
> >> >
> >> > Thanks,
> >> > Norbert
> >> >
> >> >
> >> > On Thu, Dec 20, 2018 at 9:07 AM Enrico Olivelli <eolivelli@gmail.com>
> >> wrote:
> >> >
> >> > > Great.
> >> > > Can you create JIRA tickets for remaining subtask? So that I can pick
> >> them
> >> > > up?
> >> > > I volounter for spotbugs and for CI integration, but let's see the
> >> list
> >> > > Enrico
> >> > >
> >> > > Il gio 20 dic 2018, 07:21 Andor Molnar <andor@apache.org> ha scritto:
> >> > >
> >> > > > Ok. Looks like ant still works properly, so let’s commit this patch
> >> and
> >> > > > you guys can collaborate to polish the Maven build.
> >> > > >
> >> > > > For now, it’s master-only.
> >> > > >
> >> > > > Thanks,
> >> > > > Andor
> >> > > >
> >> > > >
> >> > > >
> >> > > > > On 2018. Dec 19., at 16:44, Norbert Kalmar
> >> > > <nkalmar@cloudera.com.INVALID>
> >> > > > wrote:
> >> > > > >
> >> > > > > Thank you Enrico, I agree, that we could commit this patch at it's
> >> > > > current
> >> > > > > state, it fulfills the original jira anyways.
> >> > > > >
> >> > > > > I'll see what's wrong with the java tests, but honestly, it looks
> >> like
> >> > > > > they're just flaky... runs well on local builds with 8 thread.
> >> > > > >
> >> > > > > Regards,
> >> > > > > Norbert
> >> > > > >
> >> > > > > On Wed, Dec 19, 2018 at 2:50 PM Tamas Penzes
> >> > > <tamaas@cloudera.com.invalid
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > >> Hi All,
> >> > > > >>
> >> > > > >> For assembly task I would promote the way how HBase works.
> >> > > > >> They create a pure source and a bin tarball separately. Please
> >> see how
> >> > > > they
> >> > > > >> create a release here:
> >> > > > >>
> >> https://github.com/apache/hbase/blob/master/dev-support/make_rc.sh
> >> > > > >> We could probably use the well known "copy+paste technology" to
> >> have
> >> > > it
> >> > > > >> within ZooKeeper the same way. ;-)
> >> > > > >>
> >> > > > >> Regards, Tamaas
> >> > > > >>
> >> > > > >> On Wed, Dec 19, 2018 at 2:28 PM Enrico Olivelli <
> >> eolivelli@gmail.com>
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >>> Great work Norbert
> >> > > > >>> I you want I can help,especially for rat, findbugs (need to
> >> switch to
> >> > > > >>> spotbugs anyway) and OWASP stuff (recently I started using Maven
> >> > > > >>> Plugin in other projects)
> >> > > > >>> But I am not sure how can I help you concretely if we do not
> >> commit
> >> > > > your
> >> > > > >>> work.
> >> > > > >>> We could commit the work as it is now, leaving "ant" as official
> >> > > build
> >> > > > >>> method, but having the poms committed will ease collaboration.
> >> > > > >>>
> >> > > > >>> We will also have to work on CI jobs, I can help on that part
> >> as well
> >> > > > >>>
> >> > > > >>> Enrico
> >> > > > >>>
> >> > > > >>> Il giorno mer 19 dic 2018 alle ore 12:26 Norbert Kalmar
> >> > > > >>> <nkalmar@cloudera.com.invalid> ha scritto:
> >> > > > >>>>
> >> > > > >>>> Hi everyone,
> >> > > > >>>>
> >> > > > >>>> Some update on the maven migration: I had a few bumps here and
> >> there
> >> > > > >>> (just
> >> > > > >>>> looking at the latest patch Andor linked -
> >> > > > >>>> https://github.com/apache/zookeeper/pull/708 - you can see on
> >> the
> >> > > > >>> commits).
> >> > > > >>>> Current state is that the build works, tests run, but reports
> >> like
> >> > > > >>>> findbugs, clover etc. are not yet implemented. Maven has
> >> plugins for
> >> > > > >> them
> >> > > > >>>> usually, but it's not always trivial, especially with the C
> >> client.
> >> > > > The
> >> > > > >>>> assembly is also left to be done, but it should be fairly easy
> >> to
> >> > > do a
> >> > > > >>>> similar tarball then ant does (although this will be also an
> >> > > > >> interesting
> >> > > > >>>> task, as ant does some strange things, like duplicated sources
> >> of
> >> > > most
> >> > > > >>>> contrib projects).
> >> > > > >>>>
> >> > > > >>>> I had a seperate jira to do the recipes and contrib maven
> >> build. I
> >> > > do
> >> > > > >> not
> >> > > > >>>> have open PR for it, but recipes is done and I am now working
> >> on the
> >> > > > >>>> contrib projects. Most of them is manually build and never gets
> >> > > called
> >> > > > >>> from
> >> > > > >>>> the main build.xml. I will not integrate these either to the
> >> maven
> >> > > > >> build.
> >> > > > >>>> The reason is that there are plans to remove some of them from
> >> ZK
> >> > > repo
> >> > > > >>>> anyway. The other reason is that for starters, we want to
> >> replicate
> >> > > > the
> >> > > > >>> ant
> >> > > > >>>> build as closely as possible, without doing any nasty
> >> workarounds in
> >> > > > >>> maven
> >> > > > >>>> to achieve that. And from there, we can improve, use maven's
> >> > > > advantages
> >> > > > >>> to
> >> > > > >>>> shape the build of ZooKeeper. Once it is stable and proven to
> >> have
> >> > > all
> >> > > > >>> the
> >> > > > >>>> functionality required for build and release.
> >> > > > >>>>
> >> > > > >>>> Right now, I am trying to stabilize the build as much as
> >> possible.
> >> > > > >> Andor
> >> > > > >>>> also fixed some flaky C tests that for some strange reasons,
> >> become
> >> > > > >>>> extremely flaky with the maven build:
> >> > > > >>>> https://github.com/apache/zookeeper/pull/740
> >> > > > >>>>
> >> > > > >>>> Regards,
> >> > > > >>>> Norbert
> >> > > > >>>>
> >> > > > >>>> On Tue, Dec 18, 2018 at 9:52 AM Andor Molnar
> >> > > > >> <andor@cloudera.com.invalid
> >> > > > >>>>
> >> > > > >>>> wrote:
> >> > > > >>>>
> >> > > > >>>>> Sure, good point. Let's put it on the list.
> >> > > > >>>>>
> >> > > > >>>>> Andor
> >> > > > >>>>>
> >> > > > >>>>>
> >> > > > >>>>> On Tue, Dec 18, 2018 at 12:17 AM Patrick Hunt <
> >> phunt@apache.org>
> >> > > > >>> wrote:
> >> > > > >>>>>
> >> > > > >>>>>> Are folks OK to wait on that OWASP issue I documented over
> >> the
> >> > > > >>> weekend?
> >> > > > >>>>>> afaict we are not affected but it would be good to get
> >> another
> >> > > pair
> >> > > > >>> of
> >> > > > >>>>> eyes
> >> > > > >>>>>> on it.
> >> > > > >>>>>>
> >> > > > >>>>>> Patrick
> >> > > > >>>>>>
> >> > > > >>>>>> On Mon, Dec 17, 2018 at 2:55 PM Andor Molnár <
> >> andor@apache.org>
> >> > > > >>> wrote:
> >> > > > >>>>>>
> >> > > > >>>>>>> Hi team,
> >> > > > >>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>> I'm proudly announce that thanks to the joint effort from
> >> the
> >> > > > >>>>> community,
> >> > > > >>>>>>> the 3.5 blockers list has become empty:
> >> > > > >>>>>>>
> >> > > > >>>>>>> "project = ZooKeeper AND resolution = Unresolved AND
> >> fixVersion =
> >> > > > >>> 3.5.5
> >> > > > >>>>>>> AND priority in (blocker, critical) ORDER BY priority DESC,
> >> key
> >> > > > >>> ASC"
> >> > > > >>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>> Well... almost. All the blocker issues have gone, but we
> >> still
> >> > > > >>> have the
> >> > > > >>>>>>> Maven migration to complete before the stable release. If
> >> you
> >> > > > >> have
> >> > > > >>> some
> >> > > > >>>>>>> free cycles, please join us testing the Maven build on this
> >> PR:
> >> > > > >>>>>>>
> >> > > > >>>>>>> https://github.com/apache/zookeeper/pull/708
> >> > > > >>>>>>>
> >> > > > >>>>>>> I hope we can merge it pretty soon.
> >> > > > >>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>> In terms of the builds, the weather at 3.5 branch is quite
> >> sunny
> >> > > > >>>>>> nowadays:
> >> > > > >>>>>>>
> >> > > > >>>>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/
> >> > > > >>>>>>>
> >> > > > >>>>>>> The Java 11 build is still having some difficulties, which
> >> > > > >>> hopefully I
> >> > > > >>>>>>> can address before the holidays:
> >> > > > >>>>>>>
> >> > > > >>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-3204
> >> > > > >>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>> If you happen to know about something which is important
> >> from
> >> > > > >> 3.5's
> >> > > > >>>>>>> perspective and missing from the above, please don't
> >> hesitate to
> >> > > > >>> share.
> >> > > > >>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>> Happy ZooKeeping!
> >> > > > >>>>>>>
> >> > > > >>>>>>> Andor
> >> > > > >>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>> On 11/2/18 21:12, Fangmin Lv wrote:
> >> > > > >>>>>>>> Andor,
> >> > > > >>>>>>>>
> >> > > > >>>>>>>> Here is the PR to port ZK-3104 from master to 3.4:
> >> > > > >>>>>>>> https://github.com/apache/zookeeper/pull/685.
> >> > > > >>>>>>>>
> >> > > > >>>>>>>> Fangmin
> >> > > > >>>>>>>>
> >> > > > >>>>>>>> On Fri, Nov 2, 2018 at 11:46 AM Fangmin Lv <
> >> > > > >> lvfangmin@gmail.com>
> >> > > > >>>>>> wrote:
> >> > > > >>>>>>>>
> >> > > > >>>>>>>>> Hi Andor,
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>> Is anyone working on ZK-2778? I can pick it up if there
> >> is no
> >> > > > >>> one
> >> > > > >>>>>>> working
> >> > > > >>>>>>>>> on it yet.
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>> I'll open a 3.5 PR for ZK-3104 today.
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>> Fangmin
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>> On Fri, Oct 26, 2018 at 3:33 AM Andor Molnar <
> >> > > > >> andor@apache.org>
> >> > > > >>>>>> wrote:
> >> > > > >>>>>>>>>
> >> > > > >>>>>>>>>> Hi folks,
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> You’ve probably realised lots of update emails coming
> >> from
> >> > > > >>> Jira.
> >> > > > >>>>>> Please
> >> > > > >>>>>>>>>> be aware that we’ve updated a bunch of open
> >> blocker/critical
> >> > > > >>> 3.5
> >> > > > >>>>>>> tickets to
> >> > > > >>>>>>>>>> reflect to what we discussed in this email.
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> If you open up the following jira filter:
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> project = ZooKeeper and resolution = Unresolved and
> >> > > > >> fixVersion
> >> > > > >>> =
> >> > > > >>>>>> 3.5.5
> >> > > > >>>>>>>>>> AND priority in (blocker, critical) ORDER BY priority
> >> DESC,
> >> > > > >>> key ASC
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> You’ll see the most up-to-date list of tickets which
> >> need to
> >> > > > >> be
> >> > > > >>>>>>> addressed
> >> > > > >>>>>>>>>> before the stable 3.5 release.
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> Thank you for your efforts to get this done.
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> Fangmin, ZK-3104 is waiting for backport, but ticket has
> >> > > > >>> already
> >> > > > >>>>> been
> >> > > > >>>>>>>>>> resolved. Have you created a separate ticket for the
> >> backport
> >> > > > >>> or
> >> > > > >>>>>> shall
> >> > > > >>>>>>> I
> >> > > > >>>>>>>>>> just reopen it with the right fix versions?
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>> Thanks,
> >> > > > >>>>>>>>>> Andor
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>>>>> On 2018. Oct 8., at 12:34, Andor Molnar <
> >> andor@apache.org>
> >> > > > >>> wrote:
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>> Hi,
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>> Let me summarize and give a quick update on the
> >> outstanding
> >> > > > >>> issues
> >> > > > >>>>>> for
> >> > > > >>>>>>>>>> 3.5 GA:
> >> > > > >>>>>>>>>>> - ZOOKEEPER-1818 (Fix don't care for trunk)
> >> > > > >>>>>>>>>>> - ZOOKEEPER-2778 (Potential server deadlock between
> >> follower
> >> > > > >>> sync
> >> > > > >>>>>> with
> >> > > > >>>>>>>>>> leader and follower receiving external connection
> >> requests.)
> >> > > > >>>>>>>>>>> - ZOOKEEPER-3021 Migrate project structure to Maven
> >> > > > >> (ongoing)
> >> > > > >>>>>>>>>>> - ZOOKEEPER-925 Docs generation to Maven
> >> > > > >>>>>>>>>>> - ZOOKEEPER-3104 (waiting for backport)
> >> > > > >>>>>>>>>>> - ZOOKEEPER-3125 (waiting for backport PR #647)
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>> The 2 Maven related tickets are no-brainers as well as
> >> the
> >> > > > >>>>>> backports.
> >> > > > >>>>>>>>>> ZK-2778 has been picked up by Maoling (thanks!) as far
> >> as I
> >> > > > >> can
> >> > > > >>>>> see,
> >> > > > >>>>>>>>>> ZK-1818 is the only one waiting for a volunteer.
> >> > > > >>>>>>>>>>> Please correct me if I’ve missed something.
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>> Regards,
> >> > > > >>>>>>>>>>> Andor
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>>
> >> > > > >>>>>>>>>>>> On 2018. Sep 28., at 18:32, Tamas Penzes
> >> > > > >>>>>> <tamaas@cloudera.com.INVALID
> >> > > > >>>>>>>>
> >> > > > >>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>> Hi All,
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>> I would add ZOOKEEPER-3021
> >> > > > >>>>>>>>>>>> <https://issues.apache.org/jira/browse/ZOOKEEPER-3021>
> >> > > > >>> Migrate
> >> > > > >>>>>>> project
> >> > > > >>>>>>>>>>>> structure to Maven build as a blocker too. Since the
> >> > > > >>> migration
> >> > > > >>>>> has
> >> > > > >>>>>>>>>> started
> >> > > > >>>>>>>>>>>> it would be good to finish before releasing ZK 3.5.x
> >> GA.
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>> ZOOKEEPER-925 <
> >> > > > >>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-925
> >> > > > >>>>>>>
> >> > > > >>>>>>>>>> replace
> >> > > > >>>>>>>>>>>> our forrest site and documentation generation might
> >> also
> >> > > > >> be a
> >> > > > >>>>> good
> >> > > > >>>>>>>>>> idea,
> >> > > > >>>>>>>>>>>> since then we could deliver the new MarkDown based
> >> > > > >>> documentation.
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>> Regards, Tamaas
> >> > > > >>>>>>>>>>>>
> >> > > > >>>>>>>>>>>> On Fri, Sep 14, 2018 at 10:09 AM Fangmin Lv <
> >> > > > >>> lvfangmin@gmail.com
> >> > > > >>>>>>
> >> > > > >>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>> Oh, sorry for the confusion, I should provide more
> >> > > > >> context.
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>> Leader will use on disk txn sync with followers to if
> >> the
> >> > > > >>> peer
> >> > > > >>>>>> zxid
> >> > > > >>>>>>>>>> is not
> >> > > > >>>>>>>>>>>>> in it's in memory commit logs, the code is here:
> >> Leader on
> >> > > > >>> disk
> >> > > > >>>>>> txn
> >> > > > >>>>>>>>>> sync
> >> > > > >>>>>>>>>>>>> <
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > >
> >> https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/quorum/LearnerHandler.java#L774
> >> > > > >>>>>>>>>>>>>> .
> >> > > > >>>>>>>>>>>>> There is bug that potentially there will be gap in
> >> the txn
> >> > > > >>>>> files,
> >> > > > >>>>>>> like
> >> > > > >>>>>>>>>>>>> after snap sync, etc, so it's possible the peer will
> >> miss
> >> > > > >>> txns
> >> > > > >>>>> due
> >> > > > >>>>>>> to
> >> > > > >>>>>>>>>> this.
> >> > > > >>>>>>>>>>>>> The option to disable it is snapshotSizeFactor
> >> > > > >>>>>>>>>>>>> <
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > >
> >> https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZKDatabase.java#L81
> >> > > > >>>>>>>>>>>>>> ,
> >> > > > >>>>>>>>>>>>> set it to -1 will disable this feature. On 3.5, it's
> >> > > > >> better
> >> > > > >>> to
> >> > > > >>>>>> have
> >> > > > >>>>>>> a
> >> > > > >>>>>>>>>> PR to
> >> > > > >>>>>>>>>>>>> set this to -1 by default. It might have more SNAP
> >> sync,
> >> > > > >> but
> >> > > > >>>>> from
> >> > > > >>>>>>> our
> >> > > > >>>>>>>>>> prod
> >> > > > >>>>>>>>>>>>> it doesn't seem to be a big problem to me.
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>> I can send out the diff to disable it by default on
> >> 3.5 if
> >> > > > >>> you
> >> > > > >>>>>> guys
> >> > > > >>>>>>>>>> think
> >> > > > >>>>>>>>>>>>> this is the right way to do.
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>> Thanks,
> >> > > > >>>>>>>>>>>>> Fangmin
> >> > > > >>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>> On Thu, Sep 13, 2018 at 1:58 AM Andor Molnar <
> >> > > > >>> andor@apache.org>
> >> > > > >>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>> What’s needed to turn it off?
> >> > > > >>>>>>>>>>>>>> Do we need a PR or it’s just a config option?
> >> > > > >>>>>>>>>>>>>> Shall we implement a feature switch for that and
> >> turn it
> >> > > > >>> off by
> >> > > > >>>>>>>>>> default?
> >> > > > >>>>>>>>>>>>>> Sorry I don’t have too much insight on disk txn sync.
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>> Andor
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>> On 2018. Sep 13., at 9:16, Fangmin Lv <
> >> > > > >>> lvfangmin@gmail.com>
> >> > > > >>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>> And to be clear, ZOOKEEPER-2418 is actually just one
> >> > > > >> case
> >> > > > >>> of
> >> > > > >>>>>>>>>>>>>> inconsistency
> >> > > > >>>>>>>>>>>>>>> which could caused by on disk txn sync, as I
> >> mentioned
> >> > > > >> in
> >> > > > >>> a
> >> > > > >>>>>> newer
> >> > > > >>>>>>>>>> JIRA
> >> > > > >>>>>>>>>>>>>>> ZOOKEEPER-2846 <
> >> > > > >>>>>>>>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2846>,
> >> > > > >>>>>>>>>>>>>> the
> >> > > > >>>>>>>>>>>>>>> snap sync or txn sync could also leave txns gap in
> >> the
> >> > > > >> txn
> >> > > > >>>>> file,
> >> > > > >>>>>>>>>> which
> >> > > > >>>>>>>>>>>>>> is a
> >> > > > >>>>>>>>>>>>>>> more common case could trigger this issue.
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>> I would suggest to turn off the on disk txn sync by
> >> > > > >>> default
> >> > > > >>>>> for
> >> > > > >>>>>>> now
> >> > > > >>>>>>>>>> to
> >> > > > >>>>>>>>>>>>>>> avoid this issue, after we finished ZOOKEEPER-3114,
> >> we
> >> > > > >>> can use
> >> > > > >>>>>>> that
> >> > > > >>>>>>>>>> to
> >> > > > >>>>>>>>>>>>>>> validate the on disk txns during syncing.
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>> Thanks,
> >> > > > >>>>>>>>>>>>>>> Fangmin
> >> > > > >>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:55 AM Fangmin Lv <
> >> > > > >>>>> lvfangmin@gmail.com
> >> > > > >>>>>>>
> >> > > > >>>>>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>> Andor,
> >> > > > >>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>> ZOOKEEPER-3114 is about adding real time digest
> >> > > > >> checking
> >> > > > >>> to
> >> > > > >>>>>> help
> >> > > > >>>>>>>>>>>>>> detecting
> >> > > > >>>>>>>>>>>>>>>> inconsistency, it's a new feature with amounts of
> >> code
> >> > > > >>>>> change.
> >> > > > >>>>>>> I'll
> >> > > > >>>>>>>>>>>>>> start
> >> > > > >>>>>>>>>>>>>>>> upstream it part by part, but I don't expect it's
> >> being
> >> > > > >>>>> merged
> >> > > > >>>>>> in
> >> > > > >>>>>>>>>> the
> >> > > > >>>>>>>>>>>>>> next
> >> > > > >>>>>>>>>>>>>>>> few weeks. So yes, it's a nice to have, but
> >> definitely
> >> > > > >>> not a
> >> > > > >>>>>>> block
> >> > > > >>>>>>>>>> for
> >> > > > >>>>>>>>>>>>>> 3.5.
> >> > > > >>>>>>>>>>>>>>>> Thanks,
> >> > > > >>>>>>>>>>>>>>>> Fangmin
> >> > > > >>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 2:55 AM Andor Molnar <
> >> > > > >>>>> andor@apache.org
> >> > > > >>>>>>>
> >> > > > >>>>>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>>> Fangmin,
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> Sorry, I just noticed that you want to include the
> >> > > > >>>>> consistency
> >> > > > >>>>>>>>>> fixes
> >> > > > >>>>>>>>>>>>> in
> >> > > > >>>>>>>>>>>>>>>>> the stable version which is fine. Let’s finish the
> >> > > > >>> backports
> >> > > > >>>>>> and
> >> > > > >>>>>>>>>>>>> we’ll
> >> > > > >>>>>>>>>>>>>> be
> >> > > > >>>>>>>>>>>>>>>>> done with them.
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> ZOOKEEPER-3114 is essentially a new feature, I
> >> > > > >> wouldn’t
> >> > > > >>>>> block
> >> > > > >>>>>>> 3.5
> >> > > > >>>>>>>>>>>>> with
> >> > > > >>>>>>>>>>>>>>>>> that. What do you think?
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>> Andor
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>> On 2018. Sep 12., at 11:52, Andor Molnar <
> >> > > > >>> andor@apache.org
> >> > > > >>>>>>
> >> > > > >>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>>>> Cool, thanks for the clarification.
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>> The updated list is as follows:
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>> - ZOOKEEPER-236 (SSL/TLS support for Atomic
> >> Broadcast
> >> > > > >>>>>> protocol)
> >> > > > >>>>>>>>>>>>>>>>>> - ZOOKEEPER-1818 (Fix don't care for trunk)
> >> > > > >>>>>>>>>>>>>>>>>> - ZOOKEEPER-2778 (Potential server deadlock
> >> between
> >> > > > >>>>> follower
> >> > > > >>>>>>> sync
> >> > > > >>>>>>>>>>>>> with
> >> > > > >>>>>>>>>>>>>>>>> leader and follower receiving external connection
> >> > > > >>> requests.)
> >> > > > >>>>>>>>>>>>>>>>>> The following are not critical and no blockers
> >> for
> >> > > > >> the
> >> > > > >>>>> stable
> >> > > > >>>>>>>>>>>>> release:
> >> > > > >>>>>>>>>>>>>>>>>> Waiting for to be ported to 3.5:
> >> > > > >>>>>>>>>>>>>>>>>> - ZOOKEEPER-3104
> >> > > > >>>>>>>>>>>>>>>>>> - ZOOKEEPER-3125
> >> > > > >>>>>>>>>>>>>>>>>> - ZOOKEEPER-3127
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>> New feature:
> >> > > > >>>>>>>>>>>>>>>>>> - ZOOKEEPER-3114 (fixes ZOOKEEPER-2184 too)
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>> Regards,
> >> > > > >>>>>>>>>>>>>>>>>> Andor
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>> On 2018. Sep 12., at 0:42, Fangmin Lv <
> >> > > > >>>>> lvfangmin@gmail.com>
> >> > > > >>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>>>>> Hi Andor,
> >> > > > >>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>> That's the on disk txn feature, which was
> >> disabled
> >> > > > >>>>>> internally
> >> > > > >>>>>>>>>> after
> >> > > > >>>>>>>>>>>>>> we
> >> > > > >>>>>>>>>>>>>>>>>>> found the potentially inconsistent issue. The
> >> only
> >> > > > >>>>> solution
> >> > > > >>>>>> we
> >> > > > >>>>>>>>>> have
> >> > > > >>>>>>>>>>>>>>>>> for now
> >> > > > >>>>>>>>>>>>>>>>>>> is waiting for the new digest checking feature I
> >> > > > >>> mentioned
> >> > > > >>>>>> in
> >> > > > >>>>>>>>>>>>>>>>>>> ZOOKEEPER-3114.
> >> > > > >>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>> I think there are some other critical consistent
> >> > > > >>> issues we
> >> > > > >>>>>>> just
> >> > > > >>>>>>>>>>>>> fixed
> >> > > > >>>>>>>>>>>>>>>>> on
> >> > > > >>>>>>>>>>>>>>>>>>> master recently: ZOOKEEPER-3104, ZOOKEEPER-3125,
> >> > > > >>>>>>>>>> ZOOKEEPER-3127, I
> >> > > > >>>>>>>>>>>>>>>>> think we
> >> > > > >>>>>>>>>>>>>>>>>>> should include that in the official 3.5 release
> >> as
> >> > > > >>> well.
> >> > > > >>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > >>>>>>>>>>>>>>>>>>> Fangmin
> >> > > > >>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 11:58 AM Andor Molnár <
> >> > > > >>>>>>> andor@apache.org
> >> > > > >>>>>>>>>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>>>>>> Hi Jeelani,
> >> > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>> Thanks for letting me know. I'm happy to
> >> remove it
> >> > > > >>> from
> >> > > > >>>>> the
> >> > > > >>>>>>>>>> list
> >> > > > >>>>>>>>>>>>> to
> >> > > > >>>>>>>>>>>>>>>>> get
> >> > > > >>>>>>>>>>>>>>>>>>>> closer to a stable release. :)
> >> > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>> What's the feature which can be disabled to
> >> avoid
> >> > > > >>> data
> >> > > > >>>>>>>>>>>>>> inconsistency?
> >> > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>> Andor
> >> > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>> On 09/10/2018 11:33 PM, Mohamed Jeelani wrote:
> >> > > > >>>>>>>>>>>>>>>>>>>>> Thanks Andor for compiling this. Should we be
> >> > > > >>> ignoring
> >> > > > >>>>>>>>>>>>>>>>> ZOOKEEPER-2418 as
> >> > > > >>>>>>>>>>>>>>>>>>>> well? This exists in 3.4 as well and the
> >> feature
> >> > > > >> can
> >> > > > >>> be
> >> > > > >>>>>>>>>> disabled.
> >> > > > >>>>>>>>>>>>> We
> >> > > > >>>>>>>>>>>>>>>>> are
> >> > > > >>>>>>>>>>>>>>>>>>>> working on a longer term fix for it in 3.6.
> >> > > > >>>>>>>>>>>>>>>>>>>>> Regards,
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>> Jeelani
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>> On 9/10/18, 5:19 AM, "Andor Molnar"
> >> > > > >>>>>>>>>> <andor@cloudera.com.INVALID
> >> > > > >>>>>>>>>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>>>>>>> Fine.
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>> I'm happy to ignore 1549, 2846 and 2930.
> >> Still we
> >> > > > >>> have
> >> > > > >>>>> the
> >> > > > >>>>>>>>>> list
> >> > > > >>>>>>>>>>>>>> of:
> >> > > > >>>>>>>>>>>>>>>>>>>>> - ZOOKEEPER-236 (SSL/TLS support for Atomic
> >> > > > >>> Broadcast
> >> > > > >>>>>>>>>> protocol)
> >> > > > >>>>>>>>>>>>>>>>>>>>> - ZOOKEEPER-1818 (Fix don't care for trunk)
> >> > > > >>>>>>>>>>>>>>>>>>>>> - ZOOKEEPER-2418 (txnlog diff sync can skip
> >> > > > >> sending
> >> > > > >>> some
> >> > > > >>>>>>>>>>>>>>>>>>>> transactions to
> >> > > > >>>>>>>>>>>>>>>>>>>>> followers)
> >> > > > >>>>>>>>>>>>>>>>>>>>> - ZOOKEEPER-2778 (Potential server deadlock
> >> > > > >> between
> >> > > > >>>>>> follower
> >> > > > >>>>>>>>>>>>> sync
> >> > > > >>>>>>>>>>>>>>>>>>>> with
> >> > > > >>>>>>>>>>>>>>>>>>>>> leader and follower receiving external
> >> connection
> >> > > > >>>>>> requests.)
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>> SSL (ZK-236) is a feature which essential for
> >> the
> >> > > > >>> 3.5
> >> > > > >>>>>>> release,
> >> > > > >>>>>>>>>>>>>>>>> hence
> >> > > > >>>>>>>>>>>>>>>>>>>> I
> >> > > > >>>>>>>>>>>>>>>>>>>>> wouldn't leave it out or postpone it for the
> >> next
> >> > > > >>> stable
> >> > > > >>>>>>>>>>>>> release.
> >> > > > >>>>>>>>>>>>>>>>> PR
> >> > > > >>>>>>>>>>>>>>>>>>>> has
> >> > > > >>>>>>>>>>>>>>>>>>>>> been out for a long time, get on reviewing
> >> please.
> >> > > > >>>>>>>>>>>>>>>>>>>>> The rest are also long outstanding issues
> >> which
> >> > > > >> have
> >> > > > >>>>> been
> >> > > > >>>>>>>>>> found
> >> > > > >>>>>>>>>>>>> in
> >> > > > >>>>>>>>>>>>>>>>>>>> the 3.5
> >> > > > >>>>>>>>>>>>>>>>>>>>> branch.
> >> > > > >>>>>>>>>>>>>>>>>>>>> ZK-1818 is something which was found in 3.4
> >> and
> >> > > > >>> fixed in
> >> > > > >>>>>>> 3.4,
> >> > > > >>>>>>>>>>>>> but
> >> > > > >>>>>>>>>>>>>>>>>>>> never has
> >> > > > >>>>>>>>>>>>>>>>>>>>> been fixed in 3.5. Quite a serious issue if
> >> still
> >> > > > >>>>> present.
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>> I think we should at least run some manual
> >> testing
> >> > > > >>> and
> >> > > > >>>>> see
> >> > > > >>>>>>> if
> >> > > > >>>>>>>>>> we
> >> > > > >>>>>>>>>>>>>>>>>>>> could
> >> > > > >>>>>>>>>>>>>>>>>>>>> repro any of these issues before going ahead
> >> with
> >> > > > >> a
> >> > > > >>>>> stable
> >> > > > >>>>>>>>>>>>>> release.
> >> > > > >>>>>>>>>>>>>>>>>>>>> Regards,
> >> > > > >>>>>>>>>>>>>>>>>>>>> Andor
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>> On Fri, Sep 7, 2018 at 3:24 AM, Michael Han <
> >> > > > >>>>>>> hanm@apache.org>
> >> > > > >>>>>>>>>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>>>>>>>> I haven't went through the entire list, but
> >> looks
> >> > > > >>> like
> >> > > > >>>>>> lots
> >> > > > >>>>>>>>>> of
> >> > > > >>>>>>>>>>>>> the
> >> > > > >>>>>>>>>>>>>>>>>>>> JIRA
> >> > > > >>>>>>>>>>>>>>>>>>>>>> issues listed in this thread, such as
> >> > > > >>> ZOOKEEPER-1549,
> >> > > > >>>>>> 2846,
> >> > > > >>>>>>>>>> also
> >> > > > >>>>>>>>>>>>>>>>>>>> affects
> >> > > > >>>>>>>>>>>>>>>>>>>>>> 3.4 releases. Should we scope these issues
> >> out?
> >> > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>> I think historically the single outstanding
> >> > > > >>> blocking
> >> > > > >>>>>> issue
> >> > > > >>>>>>>>>> for a
> >> > > > >>>>>>>>>>>>>>>>>>>> stable 3.5
> >> > > > >>>>>>>>>>>>>>>>>>>>>> release is the reconfig feature and security
> >> > > > >>> concerns
> >> > > > >>>>>>> around
> >> > > > >>>>>>>>>> it
> >> > > > >>>>>>>>>>>>>>>>>>>> (somehow
> >> > > > >>>>>>>>>>>>>>>>>>>>>> addressed in ZOOKEEPER-2014), and the alpha
> >> and
> >> > > > >>> beta
> >> > > > >>>>>>> releases
> >> > > > >>>>>>>>>>>>> were
> >> > > > >>>>>>>>>>>>>>>>>>>> created
> >> > > > >>>>>>>>>>>>>>>>>>>>>> to stabilize that feature.
> >> > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > >
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__zookeeper-2Duser.578899.n2.nabble.com_Zookeeper-2Dwith-2D&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=Vl4oKanLQehvaulUvoKg8A&m=wqlhnot9c-pQLdkGkccSGNpELUNUnB-wy_h0iA3PRqI&s=_tGtL3nMWtuPrXKXDx27AIWOzyyT7W-CjIVLDFZwT0E&e=
> >> > > > >>>>>>>>>>>>>>>>>>>>>> SSL-release-date-tt7581744.html
> >> > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>> So it looks like we are in good shape to
> >> release.
> >> > > > >>>>>> Something
> >> > > > >>>>>>>>>>>>> might
> >> > > > >>>>>>>>>>>>>>>>>>>> worth
> >> > > > >>>>>>>>>>>>>>>>>>>>>> doing to claim the quality of 3.5 is on par
> >> with
> >> > > > >>> 3.4
> >> > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>> * Run Jepsen on 3.5 - 3.4 passed the test
> >> for the
> >> > > > >>>>> record
> >> > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > >
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__aphyr.com_posts_291-2Djepsen-2Dzookeeper&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=Vl4oKanLQehvaulUvoKg8A&m=wqlhnot9c-pQLdkGkccSGNpELUNUnB-wy_h0iA3PRqI&s=VjORkX5s7hrJyl8mW9Q4cfeSWF4qfTdyRjcuAiBt0y4&e=
> >> > > > >>>>>>>>>>>>>>>>>>>>>> * Fix all flaky tests on 3.5 - 3.4 has
> >> little or
> >> > > > >> no
> >> > > > >>>>> flaky
> >> > > > >>>>>>>>>> tests
> >> > > > >>>>>>>>>>>>> at
> >> > > > >>>>>>>>>>>>>>>>>>>> all.
> >> > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 4, 2018 at 1:48 AM, Andor Molnar
> >> > > > >>>>>>>>>>>>>>>>>>>> <andor@cloudera.com.invalid>
> >> > > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks Maoling! That would be huge help, I
> >> > > > >>> appreciate
> >> > > > >>>>>> it.
> >> > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>>> Andor
> >> > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>>>>>
> >> > > > >>>>>>>>>>
> >> > > > >>>>>>>
> >> > > > >>>>>>
> >> > > > >>>>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > > > --
> >> > >
> >> > >
> >> > > -- Enrico Olivelli
> >> > >
> >>
> >

Mime
View raw message