Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 941F5200B40 for ; Thu, 2 Jun 2016 07:01:58 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 92E6F160A4D; Thu, 2 Jun 2016 05:01:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1D280160A4C for ; Thu, 2 Jun 2016 07:01:56 +0200 (CEST) Received: (qmail 53888 invoked by uid 500); 2 Jun 2016 05:01:56 -0000 Mailing-List: contact dev-help@curator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@curator.apache.org Delivered-To: mailing list dev@curator.apache.org Received: (qmail 53876 invoked by uid 99); 2 Jun 2016 05:01:55 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2016 05:01:55 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 460AA180592 for ; Thu, 2 Jun 2016 05:01:55 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.78 X-Spam-Level: X-Spam-Status: No, score=0.78 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=jordanzimmerman-com.20150623.gappssmtp.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id OatVY-fX7RdF for ; Thu, 2 Jun 2016 05:01:50 +0000 (UTC) Received: from mail-yw0-f180.google.com (mail-yw0-f180.google.com [209.85.161.180]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id C1C1E5F236 for ; Thu, 2 Jun 2016 05:01:49 +0000 (UTC) Received: by mail-yw0-f180.google.com with SMTP id c127so39969438ywb.1 for ; Wed, 01 Jun 2016 22:01:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jordanzimmerman-com.20150623.gappssmtp.com; s=20150623; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=6nIdsClYIdooEcY0m4s4YKqwT/hzAIWROtaJKyADpvs=; b=aJTg+Cylma+EPp42fRRO9ujxkXMA/MphtH7Yo3ccKxH0dDLdCLN4RWEeH7l2lXHsAE 9Xk1qUeiwVKalMhTck0BbgVIf0aAfqqpbYOMbaHmVX2NafFcBUQIH22GXXvMeu3STaz6 euwgA7HymT7/H8U2RWs0dff3qKSQcR+UZETgY2ShdtoYTEwFTAtFOHh1wKQ2PW7pF57+ 9rbh0qwpxHgwyE6AljG1AmL9qp8uK/+EX6P2XxCkby2sLXl4qs6PEoAT54fNixxd3Fp3 C2H51NCWjJK7MESwou1OphrkZyAXM30M0idws4dANpvLNkBKFOs6yeGKdIELl1yDG5Rr 0xkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=6nIdsClYIdooEcY0m4s4YKqwT/hzAIWROtaJKyADpvs=; b=TAQO7qQTseoW/N3h6e1AFZGUh7ke9eRZNEwfThX80yTcXJGkJjmmosQ7joJhLBd/Te JaZR3GruZk1YtG435qch/HBuk5e8Fzx9x/G8Tdw7/pVb52j9dPB8ycQNm27zDavew86D /dghjnQSEgDeBIKxlq9/3yjk+rQGu9OGlTFgkrWEohFbH/xTxfa5CWAtYa6dtNdH5s5B f+mfPVNZETWgkF+i2priycaHyeCFn1591k/mariZHATFdcfGPUiHN4t47/mfDliHglqe X8dL6TzsySX0EyGVcx5QBm6jyl1l9WtHVGwf/pgWe3Hs/bJqTdU9WX3cgoslbcLSom5v F8FA== X-Gm-Message-State: ALyK8tLAvYVq4SBEaxuCvevHFkDE1KC8j/fBk5qG1DR+GcBBf6tTvy7OpjyDCNN0/ATc2Q== X-Received: by 10.13.232.211 with SMTP id r202mr4702547ywe.163.1464843703016; Wed, 01 Jun 2016 22:01:43 -0700 (PDT) Received: from [10.0.1.78] ([186.74.13.5]) by smtp.gmail.com with ESMTPSA id u188sm2736895ywf.8.2016.06.01.22.01.41 for (version=TLSv1/SSLv3 cipher=OTHER); Wed, 01 Jun 2016 22:01:42 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: CURATOR-3.0 tests From: Jordan Zimmerman In-Reply-To: Date: Thu, 2 Jun 2016 00:01:40 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <9BFB70F4-3965-4711-8838-B9CBC33F1BD0@jordanzimmerman.com> <18A7FA8A-6D57-4861-9740-7DFE5DBF8065@jordanzimmerman.com> <9A0A267C-D343-4E46-BE01-E1D60ECED126@jordanzimmerman.com> <053EAEE6-C5A6-4376-9734-3483A2B0E4F9@jordanzimmerman.com> <034227F5-CCE4-4D01-A5A3-5E153B3B88AB@jordanzimmerman.com> <407CDDD8-D899-415F-8B0A-60DC0642AAE3@jordanzimmerman.com> <80FD61EA-F872-48A3-8F3C-AC089FEBE236@jordanzimmerman.com> <934F96DC-62BC-4C19-A5CD-19DB8348789F@jordanzimmerman.com> <1F3FFDE0-AFA3-4AFC-81BC-156079B4259B@jordanzimmerman.com> To: dev@curator.apache.org X-Mailer: Apple Mail (2.3124) archived-at: Thu, 02 Jun 2016 05:01:58 -0000 Hmm - I=E2=80=99m still getting failures - maybe I=E2=80=99m wrong. = It=E2=80=99s late and I=E2=80=99m off to bed. I=E2=80=99ll look at this = more tomorrow. -Jordan > On Jun 1, 2016, at 10:59 PM, Cameron McKenzie = wrote: >=20 > The counter is just being used to check if semaphores are still being > acquired. Essentially it just runs in a loop acquiring semaphores (and > incrementing the counter when they are acquired). >=20 > Then it shuts down the server, waits until it the session is lost, = then > restarts the server and then checks that semaphores are being acquired > correctly again (by checking that the counter is being incremented). >=20 > This is just a simplified version of the test that is failing. >=20 > When the test fails, all of the threads are attempting to get a lease = on > the semaphore, but none of them get it, then the test times out while > waiting. >=20 >=20 >=20 > On Thu, Jun 2, 2016 at 1:29 PM, Jordan Zimmerman = > wrote: >=20 >> I also had to add: >>=20 >> while(!lost.get() && (counter.get() > 0)) >> { >> Thread.sleep(1000); >> } >> Which seems more correct to me. >>=20 >>> On Jun 1, 2016, at 9:07 PM, Cameron McKenzie = >> wrote: >>>=20 >>> I have just pushed an interprocess_mutex_issue branch. The test case = is >> in >>> TestInterprocessMutexNotReconnecting >>>=20 >>> For me it's failing around 20% of the time. >>> cheers >>>=20 >>> On Thu, Jun 2, 2016 at 11:17 AM, Cameron McKenzie < >> mckenzie.cam@gmail.com> >>> wrote: >>>=20 >>>> Yep, just let me confirm that it's actually getting the same = problem. >> I'm >>>> sure it was before, but I've just run it a bunch of times and >> everything's >>>> been fine. >>>>=20 >>>> On Thu, Jun 2, 2016 at 11:15 AM, Jordan Zimmerman < >>>> jordan@jordanzimmerman.com> wrote: >>>>=20 >>>>> Can you push your unit test somewhere? >>>>>=20 >>>>>> On Jun 1, 2016, at 7:37 PM, Cameron McKenzie = >>>>> wrote: >>>>>>=20 >>>>>> Indeed. There seems to be a problem with InterProcessSemaphoreV2 >> though. >>>>>> I've written a simplified unit test that just has a bunch of = clients >>>>>> attempting to grab a lease on the semaphore. When I shutdown and >>>>> restart ZK >>>>>> about 25% of the time, none of the clients can reacquire the >> semaphore. >>>>>>=20 >>>>>> Still trying to work out what's going on, but I'm probably not = going >> to >>>>>> have a lot of time today to look at it. >>>>>> cheers >>>>>>=20 >>>>>> On Thu, Jun 2, 2016 at 10:30 AM, Jordan Zimmerman < >>>>>> jordan@jordanzimmerman.com> wrote: >>>>>>=20 >>>>>>> Odd - SemaphoreClient does seem wrong. >>>>>>>=20 >>>>>>>> On Jun 1, 2016, at 1:43 AM, Cameron McKenzie < >> mckenzie.cam@gmail.com> >>>>>>> wrote: >>>>>>>>=20 >>>>>>>> It looks like under some circumstances (which I haven't worked = out >>>>> yet) >>>>>>>> that the InterprocessMutex acquire() is not working correctly = when >>>>>>>> reconnecting to ZK. Still digging into why this is. >>>>>>>>=20 >>>>>>>> There also seems to be a bug in the SemaphoreClient, unless I'm >>>>> missing >>>>>>>> something. At lines 126 and 140 it does compareAndSet() calls = but >>>>> throws >>>>>>> an >>>>>>>> exception if they return true. As far as I can work out, this = means >>>>> that >>>>>>>> whenever the lock is acquired, an exception gets thrown = indicating >>>>> that >>>>>>>> there are Multiple acquirers. >>>>>>>>=20 >>>>>>>> This test is failing fairly consistently. It seems to be the >> remaining >>>>>>> test >>>>>>>> that keeps failing in the Jenkins build also >>>>>>>> cheers >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> On Wed, Jun 1, 2016 at 3:10 PM, Cameron McKenzie < >>>>> mckenzie.cam@gmail.com >>>>>>>>=20 >>>>>>>> wrote: >>>>>>>>=20 >>>>>>>>> Looks like I was incorrect. The NoWatcherException is being = thrown >> on >>>>>>>>> success as well, and the problem is not in the cluster = restart. >> Will >>>>>>> keep >>>>>>>>> digging. >>>>>>>>>=20 >>>>>>>>> On Wed, Jun 1, 2016 at 2:52 PM, Cameron McKenzie < >>>>>>> mckenzie.cam@gmail.com> >>>>>>>>> wrote: >>>>>>>>>=20 >>>>>>>>>> TestInterProcessSemaphoreCluster.testCluster() is failling >>>>> (assertion >>>>>>> at >>>>>>>>>> line 294). Again, it seems like some sort of race condition = with >> the >>>>>>>>>> watcher removal. >>>>>>>>>>=20 >>>>>>>>>> When I run it in Eclipse, it fails maybe 25% of the time. = When it >>>>> fails >>>>>>>>>> it seems that it's got something to do with watcher removal. = When >>>>> the >>>>>>> test >>>>>>>>>> passes, this error is not logged. >>>>>>>>>>=20 >>>>>>>>>> org.apache.zookeeper.KeeperException$NoWatcherException: >>>>>>> KeeperErrorCode >>>>>>>>>> =3D No such watcher for /foo/bar/lock/leases >>>>>>>>>> at >>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.zookeeper.ZooKeeper$ZKWatchManager.containsWatcher(ZooKeeper.ja= va:377) >>>>>>>>>> at >>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.zookeeper.ZooKeeper$ZKWatchManager.removeWatcher(ZooKeeper.java= :252) >>>>>>>>>> at >>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.zookeeper.WatchDeregistration.unregister(WatchDeregistration.ja= va:58) >>>>>>>>>> at >> org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:712) >>>>>>>>>> at = org.apache.zookeeper.ClientCnxn.access$1500(ClientCnxn.java:97) >>>>>>>>>> at >>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:94= 8) >>>>>>>>>> at >>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:99)= >>>>>>>>>> at >>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.j= ava:361) >>>>>>>>>> at >>>>> = org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1236) >>>>>>>>>>=20 >>>>>>>>>> Is it possible it's something to do with the way that the = cluster >> is >>>>>>>>>> restarted at line 282? The old cluster is not shutdown, a new = one >> is >>>>>>> just >>>>>>>>>> created. >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> On Wed, Jun 1, 2016 at 10:44 AM, Jordan Zimmerman < >>>>>>>>>> jordan@jordanzimmerman.com> wrote: >>>>>>>>>>=20 >>>>>>>>>>> I=E2=80=99ll try to address this as part of CURATOR-333 >>>>>>>>>>>=20 >>>>>>>>>>>> On May 31, 2016, at 7:08 PM, Cameron McKenzie < >>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> Maybe we need to look at some way of providing a hook for = tests >> to >>>>>>> wait >>>>>>>>>>>> reliably for asynch tasks to finish? >>>>>>>>>>>>=20 >>>>>>>>>>>> The latest round of tests ran OK. One test failed on an >> unrelated >>>>>>> thing >>>>>>>>>>>> (ConnectionLoss), but this appears to be a transient thing = as >> it's >>>>>>>>>>> worked >>>>>>>>>>>> ok the next time around. >>>>>>>>>>>>=20 >>>>>>>>>>>> I will start getting a release together. Thanks for you = help >> with >>>>> the >>>>>>>>>>>> updated tests. >>>>>>>>>>>> cheers >>>>>>>>>>>>=20 >>>>>>>>>>>> On Wed, Jun 1, 2016 at 9:12 AM, Jordan Zimmerman < >>>>>>>>>>> jordan@jordanzimmerman.com >>>>>>>>>>>>> wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>>> The problem is in-flight watchers and async background = calls. >>>>>>> There=E2=80=99s >>>>>>>>>>> no >>>>>>>>>>>>> way to cancel these and they can take time to occur - even >> after >>>>> a >>>>>>>>>>> recipe >>>>>>>>>>>>> instance is closed. >>>>>>>>>>>>>=20 >>>>>>>>>>>>> -Jordan >>>>>>>>>>>>>=20 >>>>>>>>>>>>>> On May 31, 2016, at 5:11 PM, Cameron McKenzie < >>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Ok, running it again now. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Is the problem that the watcher clean up for the recipes = is >> done >>>>>>>>>>>>>> asynchronously after they are closed? >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> On Wed, Jun 1, 2016 at 1:35 AM, Jordan Zimmerman < >>>>>>>>>>>>> jordan@jordanzimmerman.com >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> OK - please try now. I added a loop in the =E2=80=9Cno = watchers=E2=80=9D >>>>> checker. >>>>>>> If >>>>>>>>>>>>> there >>>>>>>>>>>>>>> are remaining watchers, it will sleep a bit and try = again. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> -Jordan >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> On May 31, 2016, at 1:33 AM, Cameron McKenzie < >>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Looks like these failures are intermittent. Running = them >>>>> directly >>>>>>>>>>> in >>>>>>>>>>>>>>>> Eclipse they seem to be passing. I will run the whole = thing >>>>> again >>>>>>>>>>> in >>>>>>>>>>>>> the >>>>>>>>>>>>>>>> morning and see how it goes. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> On Tue, May 31, 2016 at 2:29 PM, Cameron McKenzie < >>>>>>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> There are still 2 tests failing for me: >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> FAILURE! - in >>>>>>>>>>>>>>>>>=20 >>>>> org.apache.curator.framework.recipes.cache.TestPathChildrenCache >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = testKilledSession(org.apache.curator.framework.recipes.cache.TestPathChild= renCache) >>>>>>>>>>>>>>>>> Time elapsed: 17.488 sec <<< FAILURE! >>>>>>>>>>>>>>>>> java.lang.AssertionError: One or more child watchers = are >>>>> still >>>>>>>>>>>>>>> registered: >>>>>>>>>>>>>>>>> [/test] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.imps.TestCleanState.closeAndTestClean(TestCle= anState.java:53) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testKille= dSession(TestPathChildrenCache.java:707) >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> FAILURE! - in >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.recipes.locks.TestInterProcessSemaphoreCluste= r >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = testCluster(org.apache.curator.framework.recipes.locks.TestInterProcessSem= aphoreCluster) >>>>>>>>>>>>>>>>> Time elapsed: 96.641 sec <<< FAILURE! >>>>>>>>>>>>>>>>> java.lang.AssertionError: expected [true] but found = [false] >>>>>>>>>>>>>>>>> at org.testng.Assert.fail(Assert.java:94) >>>>>>>>>>>>>>>>> at org.testng.Assert.failNotEquals(Assert.java:494) >>>>>>>>>>>>>>>>> at org.testng.Assert.assertTrue(Assert.java:42) >>>>>>>>>>>>>>>>> at org.testng.Assert.assertTrue(Assert.java:52) >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.recipes.locks.TestInterProcessSemaphoreCluste= r.testCluster(TestInterProcessSemaphoreCluster.java:294) >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> Failed tests: >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testKille= dSession(org.apache.curator.framework.recipes.cache.TestPathChildrenCache)= >>>>>>>>>>>>>>>>> Run 1: TestPathChildrenCache.testKilledSession:707 One = or >>>>> more >>>>>>>>>>> child >>>>>>>>>>>>>>>>> watchers are still registered: [/test] >>>>>>>>>>>>>>>>> Run 2: PASS >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> TestInterProcessSemaphoreCluster.testCluster:294 = expected >>>>> [true] >>>>>>>>>>> but >>>>>>>>>>>>>>>>> found [false] >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> Tests run: 495, Failures: 2, Errors: 0, Skipped: 0 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> On Tue, May 31, 2016 at 12:52 PM, Cameron McKenzie < >>>>>>>>>>>>>>> mckenzie.cam@gmail.com >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> Thanks, CURATOR-332 wasn't pushed. I will run the = tests >>>>> against >>>>>>>>>>> that, >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> if it's all good will merge into CURATOR-3.0 >>>>>>>>>>>>>>>>>> cheers >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> On Tue, May 31, 2016 at 12:32 PM, Jordan Zimmerman < >>>>>>>>>>>>>>>>>> jordan@jordanzimmerman.com> wrote: >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Actually - I don=E2=80=99t remember if branch = CURATOR-332 is >> merged >>>>>>>>>>> yet. I >>>>>>>>>>>>>>>>>>> made/pushed my changes in CURATOR-332 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> -jordan >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>> On May 26, 2016, at 10:04 PM, Cameron McKenzie < >>>>>>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>> I'm still seeing 6 failed tests that seem related = to the >>>>> same >>>>>>>>>>> stuff >>>>>>>>>>>>>>>>>>> after >>>>>>>>>>>>>>>>>>>> merging your fix: >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>> Failed tests: >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testBasic= s(org.apache.curator.framework.recipes.cache.TestPathChildrenCache) >>>>>>>>>>>>>>>>>>>> Run 1: TestPathChildrenCache.testBasics:863 One or = more >>>>> child >>>>>>>>>>>>>>> watchers >>>>>>>>>>>>>>>>>>>> are still registered: [/test] >>>>>>>>>>>>>>>>>>>> Run 2: TestPathChildrenCache.testBasics:863 One or = more >>>>> child >>>>>>>>>>>>>>> watchers >>>>>>>>>>>>>>>>>>>> are still registered: [/test] >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testBasic= sOnTwoCachesWithSameExecutor(org.apache.curator.framework.recipes.cache.Te= stPathChildrenCache) >>>>>>>>>>>>>>>>>>>> Run 1: >>>>>>>>>>>>>>>=20 >> TestPathChildrenCache.testBasicsOnTwoCachesWithSameExecutor:934 >>>>>>>>>>>>>>>>>>>> One or more child watchers are still registered: = [/test] >>>>>>>>>>>>>>>>>>>> Run 2: >>>>>>>>>>>>>>>=20 >> TestPathChildrenCache.testBasicsOnTwoCachesWithSameExecutor:934 >>>>>>>>>>>>>>>>>>>> One or more child watchers are still registered: = [/test] >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.recipes.cache.TestPathChildrenCache.testEnsur= ePath(org.apache.curator.framework.recipes.cache.TestPathChildrenCache) >>>>>>>>>>>>>>>>>>>> Run 1: TestPathChildrenCache.testEnsurePath:363 One = or >>>>> more >>>>>>>>>>> child >>>>>>>>>>>>>>>>>>>> watchers are still registered: [/one/two/three] >>>>>>>>>>>>>>>>>>>> Run 2: TestPathChildrenCache.testEnsurePath:363 One = or >>>>> more >>>>>>>>>>> child >>>>>>>>>>>>>>>>>>>> watchers are still registered: [/one/two/three] >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>> TestInterProcessSemaphoreCluster.testCluster:294 >> expected >>>>>>>>>>> [true] >>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>> found [false] >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.recipes.shared.TestSharedCount.testMultiClien= tVersioned(org.apache.curator.framework.recipes.shared.TestSharedCount) >>>>>>>>>>>>>>>>>>>> Run 1: PASS >>>>>>>>>>>>>>>>>>>> Run 2: TestSharedCount.testMultiClientVersioned:256 = One >> or >>>>>>> more >>>>>>>>>>>>> data >>>>>>>>>>>>>>>>>>>> watchers are still registered: [/count] >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>=20 >>>>>=20 >> = org.apache.curator.framework.recipes.shared.TestSharedCount.testSimple(org= .apache.curator.framework.recipes.shared.TestSharedCount) >>>>>>>>>>>>>>>>>>>> Run 1: TestSharedCount.testSimple:174 One or more = data >>>>>>>>>>> watchers are >>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>> registered: [/count] >>>>>>>>>>>>>>>>>>>> Run 2: TestSharedCount.testSimple:174 One or more = data >>>>>>>>>>> watchers are >>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>> registered: [/count] >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>> Tests run: 491, Failures: 6, Errors: 0, Skipped: 0 >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>> On Fri, May 27, 2016 at 3:30 AM, Jordan Zimmerman < >>>>>>>>>>>>>>>>>>>> jordan@jordanzimmerman.com> wrote: >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>> I see the problem. The fix is not simple though so = I=E2=80=99ll >>>>>>> spend >>>>>>>>>>> some >>>>>>>>>>>>>>>>>>> time on >>>>>>>>>>>>>>>>>>>>> it. The TL;DR is that exists watchers are still >> supposed >>>>> to >>>>>>>>>>> get >>>>>>>>>>>>> set >>>>>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>>>>>>> there is a KeeperException.NoNode and the code = isn=E2=80=99t >>>>>>> handling >>>>>>>>>>> it. >>>>>>>>>>>>>>> But, >>>>>>>>>>>>>>>>>>>>> while I was looking at the code I realized there = are >> some >>>>>>>>>>>>>>> significant >>>>>>>>>>>>>>>>>>>>> additional problems. Curator, here, is trying to = mirror >>>>> what >>>>>>>>>>>>>>>>>>> ZooKeeper does >>>>>>>>>>>>>>>>>>>>> internally which is insanely complicated. In = hindsight, >>>>> the >>>>>>>>>>> whole >>>>>>>>>>>>> ZK >>>>>>>>>>>>>>>>>>>>> watcher mechanism should=E2=80=99ve been decoupled = from the >>>>> mutator >>>>>>>>>>> APIs. >>>>>>>>>>>>>>>>>>> But, of >>>>>>>>>>>>>>>>>>>>> course, that=E2=80=99s easy for me to say now. >>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>> -Jordan >>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>> On May 26, 2016, at 1:10 AM, Cameron McKenzie < >>>>>>>>>>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>> Thanks Scott, >>>>>>>>>>>>>>>>>>>>>> Those tests are now passing for me. >>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>> Jordan, testNodeCache:testBasics() is failing >>>>> consistently >>>>>>>>>>> on the >>>>>>>>>>>>>>> 3.0 >>>>>>>>>>>>>>>>>>>>>> branch. It appears that this is actually = potentially a >>>>> bug >>>>>>>>>>> in the >>>>>>>>>>>>>>>>>>>>>> NodeCache. It ends up leaking a Watcher = reference. >> I've >>>>>>> had a >>>>>>>>>>>>> quick >>>>>>>>>>>>>>>>>>> look >>>>>>>>>>>>>>>>>>>>>> through, but I haven't dived in in any detail. = It's >> the >>>>>>>>>>>>>>>>>>>>>> WatcherRemovalManager stuff I think. If you've = got >> time, >>>>>>> can >>>>>>>>>>> you >>>>>>>>>>>>>>>>>>> have a >>>>>>>>>>>>>>>>>>>>>> look? If not, let me know and I'll do some more >> digging. >>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>> cheers >>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>> On Thu, May 26, 2016 at 11:47 AM, Cameron = McKenzie < >>>>>>>>>>>>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>> Thanks Scott. >>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>> Push the fix to master and merge it into 3.0. >>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>> Then I guess, I'll push new versions of 2.11 and = 3.2 >>>>> onto >>>>>>>>>>> Nexus. >>>>>>>>>>>>>>>>>>>>>>> cheers >>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>> On Thu, May 26, 2016 at 11:44 AM, Scott Blum < >>>>>>>>>>>>>>> dragonsinth@gmail.com >>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>> Alright, I have a fix, but it wants to be = applied to >>>>> both >>>>>>>>>>>>> master >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>> 3.0. >>>>>>>>>>>>>>>>>>>>>>>> Where should I push the fix? >>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 6:10 PM, Cameron = McKenzie < >>>>>>>>>>>>>>>>>>>>> mckenzie.cam@gmail.com >>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>> Thanks Scott, >>>>>>>>>>>>>>>>>>>>>>>>> If you just checkout the CURATOR-3.0 branch, = they >> are >>>>>>>>>>> failing >>>>>>>>>>>>>>>>>>> there. >>>>>>>>>>>>>>>>>>>>>>>>> cheers >>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 26, 2016 at 2:06 AM, Scott Blum < >>>>>>>>>>>>>>>>>>> dragonsinth@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>> Sure, what SHA are they failing at Cam? >>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 9:36 AM, Jordan = Zimmerman >> < >>>>>>>>>>>>>>>>>>>>>>>>>> jordan@jordanzimmerman.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>> Scott can you take a look? >>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>> -Jordan >>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>> On May 25, 2016, at 4:35 AM, Cameron = McKenzie < >>>>>>>>>>>>>>>>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>> Tree cache tests are still failing. I've = tried a >>>>> few >>>>>>>>>>> times >>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>>>>> love: >>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >> TestTreeCacheEventOrdering>TestEventOrdering.testEventOrdering:151 >>>>>>>>>>>>>>>>>>>>>>>>>>> actual 6 >>>>>>>>>>>>>>>>>>>>>>>>>>>> expected -31: >>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>> I will have a look into what's going on in = the >>>>>>> morning. >>>>>>>>>>>>> Given >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>>>>>>>>>>>>> may take some messing about to fix up, do = we >> just >>>>>>> want >>>>>>>>>>> to >>>>>>>>>>>>>>> vote >>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>> 2.11.0 >>>>>>>>>>>>>>>>>>>>>>>>>>>> separately, as that is all ready to go? >>>>>>>>>>>>>>>>>>>>>>>>>>>> cheers >>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 5:34 PM, Jordan >> Zimmerman >>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>> jordan@jordanzimmerman.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Great news. Thanks. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jordan Zimmerman >>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On May 25, 2016, at 12:37 AM, Cameron >> McKenzie < >>>>>>>>>>>>>>>>>>>>>>>>>> mckenzie.cam@gmail.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have fixed up the test case, all good = now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 1:45 PM, Cameron >>>>> McKenzie < >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looks like it was introduced with the = schema >>>>>>>>>>> validation >>>>>>>>>>>>>>>>>>> stuff. >>>>>>>>>>>>>>>>>>>>>>>> It >>>>>>>>>>>>>>>>>>>>>>>>>> now >>>>>>>>>>>>>>>>>>>>>>>>>>>>> does >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an ACL check prior to the backgrounding = call. >>>>>>>>>>> Because >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> unit >>>>>>>>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>>>>>>>>>>>> uses a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bogus ACL provider it just throws an >> exception >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> final String adjustedPath =3D >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> = adjustPath(client.fixForNamespace(givenPath, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> createMode.isSequential())); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> List aclList =3D >>>>>>>>>>> acling.getAclList(adjustedPath); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>> = client.getSchemaSet().getSchema(givenPath).validateCreate(createMode, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> data, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aclList); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> String returnPath =3D null; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if ( backgrounding.inBackground() ) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pathInBackground(adjustedPath, data, >>>>>>>>>>> givenPath); >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I guess the answer is to get the = test to >>>>>>> force a >>>>>>>>>>>>>>> failure >>>>>>>>>>>>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different way. With the >> UnhandledErrorListener, >>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> expectation is >>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this only gets called on backgrounding >>>>> operations? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cheers >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 1:39 PM, Cameron >>>>> McKenzie >>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Same on the master branch, but it = passes >>>>> there, >>>>>>> so >>>>>>>>>>>>> maybe >>>>>>>>>>>>>>>>>>>>>>>>> something >>>>>>>>>>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> legitimately broken the test. Will let = you >>>>> know >>>>>>> if >>>>>>>>>>> I >>>>>>>>>>>>> get >>>>>>>>>>>>>>>>>>>>>>>> stuck. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cheers >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 25, 2016 at 1:36 PM, Jordan >>>>>>> Zimmerman < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jordan@jordanzimmerman.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let me know if you need help. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It might be a bad merge. Have you = compared >>>>> it to >>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> master >>>>>>>>>>>>>>>>>>>>>>>>>> branch? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -JZ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On May 24, 2016, at 10:31 PM, = Cameron >>>>>>> McKenzie < >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mckenzie.cam@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guys, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There's a test >>>>>>>>>>>>>>> TestFrameworkBackground:testErrorListener >>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistently on the CURATOR-3.0 = branch. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can't see how it has ever worked. = It >>>>> seems to >>>>>>>>>>> try >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>> provoke >>>>>>>>>>>>>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> error >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> via a bad ACL provider. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But this ACL provider is called by = the >>>>>>>>>>>>>>> CreateBuilderImpl >>>>>>>>>>>>>>>>>>>>>>>> prior >>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> backgrounding call, which means that = the >>>>>>>>>>> exception >>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>> throws >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> happens >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the main Thread of the unit test. = So, >> it >>>>>>> just >>>>>>>>>>>>> throws >>>>>>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedOperationException which = is >>>>>>>>>>> propogated up >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point the unit test fails. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I will look at fixing this, but I = just >>>>>>> don't >>>>>>>>>>>>>>>>>>> understand >>>>>>>>>>>>>>>>>>>>>>>> how >>>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ever >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worked? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cheers >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>=20 >>>>>=20 >>>>=20 >>=20 >>=20