Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 781DA200CD7 for ; Tue, 1 Aug 2017 21:58:38 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 86052167C6C; Tue, 1 Aug 2017 19:58:35 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 34CA1167C74 for ; Tue, 1 Aug 2017 21:58:33 +0200 (CEST) Received: (qmail 95284 invoked by uid 500); 1 Aug 2017 19:58:32 -0000 Mailing-List: contact commits-help@bookkeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: bookkeeper-dev@bookkeeper.apache.org Delivered-To: mailing list commits@bookkeeper.apache.org Received: (qmail 95246 invoked by uid 99); 1 Aug 2017 19:58:32 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Aug 2017 19:58:32 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id 161D6814F3; Tue, 1 Aug 2017 19:58:29 +0000 (UTC) Date: Tue, 01 Aug 2017 19:58:29 +0000 To: "commits@bookkeeper.apache.org" Subject: [bookkeeper] branch master updated: ISSUE #339: Use ephemeral port for bookkeeper tests and variable unit test improvements MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <150161750921.18849.11614268313267855561@gitbox.apache.org> From: sijie@apache.org Reply-To: "commits@bookkeeper.apache.org" X-Git-Host: gitbox.apache.org X-Git-Repo: bookkeeper X-Git-Refname: refs/heads/master X-Git-Reftype: branch X-Git-Oldrev: b299a699794c1f0aa8447704795361739a8bd055 X-Git-Newrev: 3daf94143ba2b1352a56c68e1b886d7d7cd499a5 X-Git-Rev: 3daf94143ba2b1352a56c68e1b886d7d7cd499a5 X-Git-NotificationType: ref_changed_plus_diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated archived-at: Tue, 01 Aug 2017 19:58:38 -0000 This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/bookkeeper.git The following commit(s) were added to refs/heads/master by this push: new 3daf941 ISSUE #339: Use ephemeral port for bookkeeper tests and variable unit test improvements 3daf941 is described below commit 3daf94143ba2b1352a56c68e1b886d7d7cd499a5 Author: Jia Zhai AuthorDate: Tue Aug 1 12:58:19 2017 -0700 ISSUE #339: Use ephemeral port for bookkeeper tests and variable unit test improvements Descriptions of the changes in this PR: - Enable ephemeral port by default for all BookKeeperClusterTestCase based test cases - Removed Multi* based classes. A lot of unit tests unnecessarily extended from those base classes. Converted them to extend BookKeeperClusterTestCase directly. Confirmed this with sijie - Reduce sleep period for some test cases LedgerEntry fixes for #317 - use checkState rather than checkNull to keep the behavior consistent as before - fix BookKeeperTest#testReadEntryReleaseByteBufs TestBackwardCompat: - removed testCompact400 (400 has been very long time ago, no one is running 400) - BookKeeperNettyServer should shut down netty server when failed to construct a bookie TestDiskChecker: - increased the threshold to 0.99f. be consistent with TestBKConfiguration BookieInitializationTest - make each test method using different ledgers path, so registration won't be conflicting between tests BookieJournalRollingTest - make sure bookie port is used correctly on restarting bookies. Author: Jia Zhai Reviewers: Enrico Olivelli , Sijie Guo This closes #340 from zhaijack/use_ephemral_ports, closes #339 --- .../org/apache/bookkeeper/client/LedgerEntry.java | 10 +- .../org/apache/bookkeeper/proto/BookieServer.java | 14 +- .../bookie/BookieInitializationTest.java | 15 +- .../bookkeeper/bookie/BookieShutdownTest.java | 4 +- .../apache/bookkeeper/client/BookKeeperTest.java | 92 +++++----- .../bookkeeper/client/BookieWriteLedgerTest.java | 16 +- ...BookieWriteLedgersWithDifferentDigestsTest.java | 21 ++- .../bookkeeper/client/LedgerRecoveryTest.java | 13 +- .../apache/bookkeeper/client/ListLedgersTest.java | 16 +- .../bookkeeper/client/LocalBookKeeperTest.java | 67 -------- .../client/TestReadLastConfirmedAndEntry.java | 1 + .../bookkeeper/client/TestSpeculativeRead.java | 12 +- .../bookkeeper/conf/TestBKConfiguration.java | 2 + .../bookkeeper/proto/NetworkLessBookieTest.java | 34 +--- .../replication/AuditorLedgerCheckerTest.java | 13 +- .../replication/BookieLedgerIndexTest.java | 12 +- .../replication/TestReplicationWorker.java | 11 +- .../apache/bookkeeper/test/AsyncLedgerOpsTest.java | 13 +- .../org/apache/bookkeeper/test/BaseTestCase.java | 49 ------ .../bookkeeper/test/BookKeeperClusterTestCase.java | 47 ++++-- .../apache/bookkeeper/test/BookieFailureTest.java | 13 +- .../bookkeeper/test/BookieJournalRollingTest.java | 9 +- .../bookkeeper/test/BookieReadWriteTest.java | 14 +- .../java/org/apache/bookkeeper/test/CloseTest.java | 14 +- .../apache/bookkeeper/test/ConditionalSetTest.java | 10 +- .../apache/bookkeeper/test/LedgerDeleteTest.java | 12 +- .../bookkeeper/test/LocalBookiesRegistryTest.java | 24 +-- .../MultiLedgerManagerMultiDigestTestCase.java | 66 -------- .../test/MultiLedgerManagerTestCase.java | 56 ------- .../apache/bookkeeper/test/TestBackwardCompat.java | 186 +-------------------- .../org/apache/bookkeeper/test/ZooKeeperUtil.java | 31 ++-- .../apache/bookkeeper/util/TestDiskChecker.java | 6 +- 32 files changed, 250 insertions(+), 653 deletions(-) diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerEntry.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerEntry.java index 3187127..24d2ff4 100644 --- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerEntry.java +++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerEntry.java @@ -61,9 +61,10 @@ public class LedgerEntry { * the internal ByteBuf * * @return the content of the entry + * @throws IllegalStateException if this method is called twice */ public byte[] getEntry() { - Preconditions.checkNotNull(data, "entry content can be accessed only once"); + Preconditions.checkState(null != data, "entry content can be accessed only once"); byte[] entry = new byte[data.readableBytes()]; data.readBytes(entry); data.release(); @@ -78,9 +79,10 @@ public class LedgerEntry { * method of the returned InputStream * * @return an InputStream which gives access to the content of the entry + * @throws IllegalStateException if this method is called twice */ public InputStream getEntryInputStream() { - Preconditions.checkNotNull(data, "entry content can be accessed only once"); + Preconditions.checkState(null != data, "entry content can be accessed only once"); ByteBufInputStream res = new ByteBufInputStream(data); data = null; return res; @@ -94,8 +96,12 @@ public class LedgerEntry { * @return a ByteBuf which contains the data * * @see ClientConfiguration#setNettyUsePooledBuffers(boolean) + * @throws IllegalStateException if the entry has been retrieved by {@link #getEntry()} + * or {@link #getEntryInputStream()}. */ public ByteBuf getEntryBuffer() { + Preconditions.checkState(null != data, "entry content has been retrieved" + + " by #getEntry or #getEntryInputStream"); return data; } } diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServer.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServer.java index 8335055..d764126 100644 --- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServer.java +++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServer.java @@ -20,13 +20,13 @@ */ package org.apache.bookkeeper.proto; +import com.google.common.annotations.VisibleForTesting; import java.io.File; import java.io.IOException; +import java.lang.Integer; import java.net.MalformedURLException; import java.net.UnknownHostException; -import java.lang.Integer; import java.util.Arrays; - import org.apache.bookkeeper.bookie.Bookie; import org.apache.bookkeeper.bookie.ReadOnlyBookie; import org.apache.bookkeeper.bookie.BookieCriticalThread; @@ -53,8 +53,6 @@ import org.apache.zookeeper.KeeperException; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import com.google.common.annotations.VisibleForTesting; - import static org.apache.bookkeeper.bookie.BookKeeperServerStats.BOOKIE_SCOPE; import static org.apache.bookkeeper.bookie.BookKeeperServerStats.SERVER_SCOPE; import static org.apache.bookkeeper.replication.ReplicationStats.REPLICATION_SCOPE; @@ -95,7 +93,13 @@ public class BookieServer { this.conf = conf; this.statsLogger = statsLogger; this.nettyServer = new BookieNettyServer(this.conf, null); - this.bookie = newBookie(conf); + try { + this.bookie = newBookie(conf); + } catch (IOException | KeeperException | InterruptedException | BookieException e) { + // interrupted on constructing a bookie + this.nettyServer.shutdown(); + throw e; + } this.requestProcessor = new BookieRequestProcessor(conf, bookie, statsLogger.scope(SERVER_SCOPE)); this.nettyServer.setRequestProcessor(this.requestProcessor); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieInitializationTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieInitializationTest.java index 89404b6..a410e37 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieInitializationTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieInitializationTest.java @@ -29,7 +29,6 @@ import java.io.File; import java.io.IOException; import java.net.BindException; import java.net.InetAddress; - import org.apache.bookkeeper.bookie.BookieException.DiskPartitionDuplicationException; import org.apache.bookkeeper.bookie.LedgerDirsManager.NoWritableLedgerDirException; import org.apache.bookkeeper.client.BookKeeper; @@ -51,7 +50,9 @@ import org.apache.bookkeeper.zookeeper.ZooKeeperClient; import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.data.Stat; import org.junit.Assert; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.TestName; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -62,8 +63,20 @@ public class BookieInitializationTest extends BookKeeperClusterTestCase { private static final Logger LOG = LoggerFactory .getLogger(BookieInitializationTest.class); + @Rule + public final TestName runtime = new TestName(); + public BookieInitializationTest() { super(0); + String ledgersPath = "/" + runtime.getMethodName(); + baseClientConf.setZkLedgersRootPath(ledgersPath); + baseConf.setZkLedgersRootPath(ledgersPath); + } + + @Override + public void setUp() throws Exception { + super.setUp(); + zkUtil.createBKEnsemble("/" + runtime.getMethodName()); } private static class MockBookie extends Bookie { diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieShutdownTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieShutdownTest.java index 1746043..5f34144 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieShutdownTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieShutdownTest.java @@ -24,7 +24,6 @@ import java.nio.ByteBuffer; import java.util.Random; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; - import org.apache.bookkeeper.client.BKException; import org.apache.bookkeeper.client.LedgerHandle; import org.apache.bookkeeper.client.AsyncCallback.AddCallback; @@ -43,6 +42,7 @@ public class BookieShutdownTest extends BookKeeperClusterTestCase { public BookieShutdownTest() { super(3); + baseConf.setAllowEphemeralPorts(false); } private LedgerHandle lh; @@ -61,7 +61,7 @@ public class BookieShutdownTest extends BookKeeperClusterTestCase { */ @Test(timeout = 150000) public void testBookieRestartContinuously() throws Exception { - for (int index = 0; index < 100; index++) { + for (int index = 0; index < 10; index++) { SyncObj sync = new SyncObj(); try { // Create a ledger diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookKeeperTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookKeeperTest.java index 92ba078..7656b9b 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookKeeperTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookKeeperTest.java @@ -1,46 +1,40 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + * + */ package org.apache.bookkeeper.client; import io.netty.util.IllegalReferenceCountException; import java.util.Collections; import java.util.Enumeration; - -/* -* -* Licensed to the Apache Software Foundation (ASF) under one -* or more contributor license agreements. See the NOTICE file -* distributed with this work for additional information -* regarding copyright ownership. The ASF licenses this file -* to you under the Apache License, Version 2.0 (the -* "License"); you may not use this file except in compliance -* with the License. You may obtain a copy of the License at -* -* http://www.apache.org/licenses/LICENSE-2.0 -* -* Unless required by applicable law or agreed to in writing, -* software distributed under the License is distributed on an -* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -* KIND, either express or implied. See the License for the -* specific language governing permissions and limitations -* under the License. -* -*/ - import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicBoolean; - import org.apache.bookkeeper.conf.ClientConfiguration; import org.apache.bookkeeper.client.AsyncCallback.AddCallback; import org.apache.bookkeeper.client.AsyncCallback.ReadCallback; import org.apache.bookkeeper.client.BKException.BKBookieHandleNotAvailableException; import org.apache.bookkeeper.client.BookKeeper.DigestType; -import org.apache.bookkeeper.test.BaseTestCase; +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.apache.zookeeper.ZooKeeper; -import org.apache.zookeeper.Watcher; -import org.apache.zookeeper.WatchedEvent; import org.apache.zookeeper.KeeperException; - import org.junit.Assert; import org.junit.Test; import org.slf4j.Logger; @@ -51,15 +45,14 @@ import static org.junit.Assert.*; /** * Tests of the main BookKeeper client */ -public class BookKeeperTest extends BaseTestCase { +public class BookKeeperTest extends BookKeeperClusterTestCase { private final static Logger LOG = LoggerFactory.getLogger(BookKeeperTest.class); - DigestType digestType; + private final DigestType digestType; - public BookKeeperTest(DigestType digestType) { + public BookKeeperTest() { super(4); - - this.digestType = digestType; + this.digestType = DigestType.CRC32; } @Test @@ -69,7 +62,7 @@ public class BookKeeperTest extends BaseTestCase { .setZkTimeout(20000); CountDownLatch l = new CountDownLatch(1); - zkUtil.sleepServer(5, l); + zkUtil.sleepServer(200, TimeUnit.MILLISECONDS, l); l.await(); BookKeeper bkc = new BookKeeper(conf); @@ -84,15 +77,13 @@ public class BookKeeperTest extends BaseTestCase { .setZkTimeout(20000); CountDownLatch l = new CountDownLatch(1); - zkUtil.sleepServer(5, l); + zkUtil.sleepServer(200, TimeUnit.MILLISECONDS, l); l.await(); - ZooKeeper zk = new ZooKeeper(zkUtil.getZooKeeperConnectString(), 10000, - new Watcher() { - @Override - public void process(WatchedEvent event) { - } - }); + ZooKeeper zk = new ZooKeeper( + zkUtil.getZooKeeperConnectString(), + 50, + event -> {}); assertFalse("ZK shouldn't have connected yet", zk.getState().isConnected()); try { BookKeeper bkc = new BookKeeper(conf, zk); @@ -728,11 +719,10 @@ public class BookKeeperTest extends BaseTestCase { LedgerEntry entry = readEntries.nextElement(); assertTrue(entry.data.getClass().getName(), entry.data.getClass().getName().contains("PooledNonRetainedSlicedByteBuf")); - assertTrue(entry.data.release()); try { entry.data.release(); - fail("ByteBuf already released"); } catch (IllegalReferenceCountException ok) { + fail("ByteBuf already released"); } } } @@ -749,14 +739,12 @@ public class BookKeeperTest extends BaseTestCase { for (Enumeration readEntries = lh.readEntries(0, numEntries - 1); readEntries.hasMoreElements();) { LedgerEntry entry = readEntries.nextElement(); - // ButeBufs no reference counter - assertTrue(entry.data.release()); assertTrue(entry.data.getClass().getName(), entry.data.getClass().getName().contains("UnpooledSlicedByteBuf")); try { entry.data.release(); + } catch (IllegalReferenceCountException e) { fail("ByteBuf already released"); - } catch (IllegalReferenceCountException ok) { } } } @@ -773,12 +761,12 @@ public class BookKeeperTest extends BaseTestCase { for (Enumeration readEntries = lh.readEntries(0, numEntries - 1); readEntries.hasMoreElements();) { LedgerEntry entry = readEntries.nextElement(); - // ButeBufs not reference counter assertTrue(entry.data.getClass().getName(), entry.data.getClass().getName().contains("UnpooledSlicedByteBuf")); - assertTrue(entry.data.release()); + assertTrue("Can't release entry " + entry.getEntryId() + ": ref = " + entry.data.refCnt(), + entry.data.release()); try { - entry.data.release(); + assertFalse(entry.data.release()); fail("ByteBuf already released"); } catch (IllegalReferenceCountException ok) { } @@ -801,12 +789,14 @@ public class BookKeeperTest extends BaseTestCase { // ButeBufs not reference counter assertTrue(entry.data.getClass().getName(), entry.data.getClass().getName().contains("UnpooledSlicedByteBuf")); - assertTrue(entry.data.release()); + assertTrue("Can't release entry " + entry.getEntryId() + ": ref = " + entry.data.refCnt(), + entry.data.release()); try { - entry.data.release(); + assertFalse(entry.data.release()); fail("ByteBuf already released"); } catch (IllegalReferenceCountException ok) { - } } + } + } } } diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieWriteLedgerTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieWriteLedgerTest.java index e226bb0..f7bcffd 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieWriteLedgerTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieWriteLedgerTest.java @@ -20,10 +20,11 @@ */ package org.apache.bookkeeper.client; -import io.netty.buffer.Unpooled; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; + +import io.netty.buffer.Unpooled; import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.Enumeration; @@ -34,12 +35,11 @@ import java.util.UUID; import java.util.HashMap; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ExecutionException; - import org.apache.bookkeeper.client.AsyncCallback.AddCallback; import org.apache.bookkeeper.client.BookKeeper.DigestType; import org.apache.bookkeeper.meta.LongHierarchicalLedgerManagerFactory; import org.apache.bookkeeper.net.BookieSocketAddress; -import org.apache.bookkeeper.test.MultiLedgerManagerMultiDigestTestCase; +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.junit.Before; import org.junit.Test; import org.slf4j.Logger; @@ -49,7 +49,7 @@ import org.slf4j.LoggerFactory; * Testing ledger write entry cases */ public class BookieWriteLedgerTest extends - MultiLedgerManagerMultiDigestTestCase implements AddCallback { + BookKeeperClusterTestCase implements AddCallback { private final static Logger LOG = LoggerFactory .getLogger(BookieWriteLedgerTest.class); @@ -65,7 +65,7 @@ public class BookieWriteLedgerTest extends ArrayList entries1; // generated entries ArrayList entries2; // generated entries - DigestType digestType; + private final DigestType digestType; private static class SyncObj { volatile int counter; @@ -86,10 +86,10 @@ public class BookieWriteLedgerTest extends entries2 = new ArrayList(); // initialize the entries list } - public BookieWriteLedgerTest(String ledgerManagerFactory, - DigestType digestType) { + public BookieWriteLedgerTest() { super(5); - this.digestType = digestType; + this.digestType = DigestType.CRC32; + String ledgerManagerFactory = "org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory"; // set ledger manager baseConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); baseClientConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieWriteLedgersWithDifferentDigestsTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieWriteLedgersWithDifferentDigestsTest.java index 3b28db0..c987cf8 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieWriteLedgersWithDifferentDigestsTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieWriteLedgersWithDifferentDigestsTest.java @@ -22,14 +22,17 @@ package org.apache.bookkeeper.client; import java.nio.ByteBuffer; import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; import java.util.Enumeration; import java.util.Random; - import org.apache.bookkeeper.client.AsyncCallback.AddCallback; import org.apache.bookkeeper.client.BookKeeper.DigestType; -import org.apache.bookkeeper.test.MultiLedgerManagerMultiDigestTestCase; +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.junit.Before; import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -40,8 +43,9 @@ import static org.junit.Assert.*; * This can happen as result of clients using different settings * yet reading each other data or configuration change roll out. */ +@RunWith(Parameterized.class) public class BookieWriteLedgersWithDifferentDigestsTest extends - MultiLedgerManagerMultiDigestTestCase implements AddCallback { + BookKeeperClusterTestCase implements AddCallback { private final static Logger LOG = LoggerFactory .getLogger(BookieWriteLedgersWithDifferentDigestsTest.class); @@ -57,7 +61,7 @@ public class BookieWriteLedgersWithDifferentDigestsTest extends ArrayList entries1; // generated entries ArrayList entries2; // generated entries - DigestType digestType; + private final DigestType digestType; private static class SyncObj { volatile int counter; @@ -68,6 +72,11 @@ public class BookieWriteLedgersWithDifferentDigestsTest extends } } + @Parameterized.Parameters + public static Collection configs() { + return Arrays.asList(new Object[][] { {DigestType.MAC }, {DigestType.CRC32}}); + } + @Override @Before public void setUp() throws Exception { @@ -78,10 +87,10 @@ public class BookieWriteLedgersWithDifferentDigestsTest extends entries2 = new ArrayList(); // initialize the entries list } - public BookieWriteLedgersWithDifferentDigestsTest(String ledgerManagerFactory, - DigestType digestType) { + public BookieWriteLedgersWithDifferentDigestsTest(DigestType digestType) { super(3); this.digestType = digestType; + String ledgerManagerFactory = "org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory"; // set ledger manager baseConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); baseClientConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LedgerRecoveryTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LedgerRecoveryTest.java index 2b39eaf..5674b5d 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LedgerRecoveryTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LedgerRecoveryTest.java @@ -21,14 +21,12 @@ package org.apache.bookkeeper.client; import io.netty.buffer.ByteBuf; - import java.io.IOException; import java.util.Enumeration; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; - import org.apache.bookkeeper.bookie.Bookie; import org.apache.bookkeeper.bookie.BookieException; import org.apache.bookkeeper.client.AsyncCallback.AddCallback; @@ -39,7 +37,7 @@ import org.apache.bookkeeper.net.BookieSocketAddress; import org.apache.bookkeeper.proto.BookieProtocol; import org.apache.bookkeeper.proto.BookkeeperInternalCallbacks; import org.apache.bookkeeper.proto.BookkeeperInternalCallbacks.WriteCallback; -import org.apache.bookkeeper.test.BaseTestCase; +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -49,14 +47,15 @@ import static org.junit.Assert.*; /** * This unit test tests ledger recovery. */ -public class LedgerRecoveryTest extends BaseTestCase { +public class LedgerRecoveryTest extends BookKeeperClusterTestCase { private final static Logger LOG = LoggerFactory.getLogger(LedgerRecoveryTest.class); - DigestType digestType; + private final DigestType digestType; - public LedgerRecoveryTest(DigestType digestType) { + public LedgerRecoveryTest() { super(3); - this.digestType = digestType; + this.digestType = DigestType.CRC32; + this.baseConf.setAllowEphemeralPorts(false); } private void testInternal(int numEntries) throws Exception { diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/ListLedgersTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/ListLedgersTest.java index 8231ed8..38f02ce 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/ListLedgersTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/ListLedgersTest.java @@ -14,28 +14,22 @@ * License for the specific language governing permissions and limitations under * the License. */ - package org.apache.bookkeeper.client; import java.util.Iterator; - import org.apache.bookkeeper.client.BookKeeper.DigestType; import org.apache.bookkeeper.conf.ClientConfiguration; -import org.apache.bookkeeper.test.BaseTestCase; - +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.junit.Assert; import org.junit.Test; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -public class ListLedgersTest extends BaseTestCase { - private final static Logger LOG = LoggerFactory.getLogger(ListLedgersTest.class); +public class ListLedgersTest extends BookKeeperClusterTestCase { - DigestType digestType; + private final DigestType digestType; - public ListLedgersTest (DigestType digestType) { + public ListLedgersTest () { super(4); - this.digestType = digestType; + this.digestType = DigestType.CRC32; } @Test(timeout=60000) diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LocalBookKeeperTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LocalBookKeeperTest.java deleted file mode 100644 index 4289476..0000000 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/LocalBookKeeperTest.java +++ /dev/null @@ -1,67 +0,0 @@ -/* -* -* Licensed to the Apache Software Foundation (ASF) under one -* or more contributor license agreements. See the NOTICE file -* distributed with this work for additional information -* regarding copyright ownership. The ASF licenses this file -* to you under the Apache License, Version 2.0 (the -* "License"); you may not use this file except in compliance -* with the License. You may obtain a copy of the License at -* -* http://www.apache.org/licenses/LICENSE-2.0 -* -* Unless required by applicable law or agreed to in writing, -* software distributed under the License is distributed on an -* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -* KIND, either express or implied. See the License for the -* specific language governing permissions and limitations -* under the License. -* -*/ - -package org.apache.bookkeeper.client; - -import java.util.concurrent.CountDownLatch; - -import org.apache.bookkeeper.conf.ClientConfiguration; -import org.apache.bookkeeper.client.BookKeeper.DigestType; -import org.apache.bookkeeper.conf.ServerConfiguration; -import org.apache.bookkeeper.test.BaseTestCase; -import org.junit.Test; - -/** - * Tests of the main BookKeeper client using networkless comunication - */ -public class LocalBookKeeperTest extends BaseTestCase { - - protected ServerConfiguration newServerConfiguration() throws Exception { - return super - .newServerConfiguration() - .setEnableLocalTransport(true); - } - - DigestType digestType; - - public LocalBookKeeperTest(DigestType digestType) { - super(4); - this.digestType=digestType; - } - - @Test - public void testUseLocalBookie() throws Exception { - ClientConfiguration conf = new ClientConfiguration() - .setZkServers(zkUtil.getZooKeeperConnectString()) - .setZkTimeout(20000); - - CountDownLatch l = new CountDownLatch(1); - zkUtil.sleepServer(5, l); - l.await(); - - BookKeeper bkc = new BookKeeper(conf); - LedgerHandle h = bkc.createLedger(1,1,digestType, "testPasswd".getBytes()); - h.addEntry("test".getBytes()); - h.close(); - bkc.close(); - } - -} diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestReadLastConfirmedAndEntry.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestReadLastConfirmedAndEntry.java index 96c5c08..18c800c 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestReadLastConfirmedAndEntry.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestReadLastConfirmedAndEntry.java @@ -51,6 +51,7 @@ public class TestReadLastConfirmedAndEntry extends BookKeeperClusterTestCase { public TestReadLastConfirmedAndEntry() { super(3); this.digestType = BookKeeper.DigestType.CRC32; + this.baseConf.setAllowEphemeralPorts(false); } static class FakeBookie extends Bookie { diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestSpeculativeRead.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestSpeculativeRead.java index c0b8e49..89a1baa 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestSpeculativeRead.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestSpeculativeRead.java @@ -25,13 +25,11 @@ import java.util.BitSet; import java.util.Enumeration; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; - import org.apache.bookkeeper.conf.ClientConfiguration; import org.apache.bookkeeper.client.AsyncCallback.ReadCallback; import org.apache.bookkeeper.client.BookKeeper.DigestType; import org.apache.bookkeeper.net.BookieSocketAddress; -import org.apache.bookkeeper.test.BaseTestCase; - +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -42,15 +40,15 @@ import static org.junit.Assert.*; * This unit test tests ledger fencing; * */ -public class TestSpeculativeRead extends BaseTestCase { +public class TestSpeculativeRead extends BookKeeperClusterTestCase { private final static Logger LOG = LoggerFactory.getLogger(TestSpeculativeRead.class); - DigestType digestType; + private final DigestType digestType; byte[] passwd = "specPW".getBytes(); - public TestSpeculativeRead(DigestType digestType) { + public TestSpeculativeRead() { super(10); - this.digestType = digestType; + this.digestType = DigestType.CRC32; } long getLedgerToRead(int ensemble, int quorum) throws Exception { diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/conf/TestBKConfiguration.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/conf/TestBKConfiguration.java index bdd9e4a..fe5fb9a 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/conf/TestBKConfiguration.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/conf/TestBKConfiguration.java @@ -29,6 +29,8 @@ public class TestBKConfiguration { // enable journal format version confReturn.setJournalFormatVersionToWrite(5); confReturn.setAllowLoopback(true); + confReturn.setAllowEphemeralPorts(true); + confReturn.setBookiePort(0); confReturn.setGcWaitTime(1000); confReturn.setDiskUsageThreshold(0.999f); confReturn.setDiskUsageWarnThreshold(0.99f); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/NetworkLessBookieTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/NetworkLessBookieTest.java index 4921af9..c7e9514 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/NetworkLessBookieTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/NetworkLessBookieTest.java @@ -21,37 +21,25 @@ package org.apache.bookkeeper.proto; -import java.util.concurrent.CountDownLatch; +import io.netty.channel.Channel; +import io.netty.channel.local.LocalChannel; import org.apache.bookkeeper.client.BookKeeper; - import org.apache.bookkeeper.conf.ClientConfiguration; import org.apache.bookkeeper.client.BookKeeper.DigestType; import org.apache.bookkeeper.client.LedgerHandle; -import org.apache.bookkeeper.conf.ServerConfiguration; -import org.apache.bookkeeper.test.BaseTestCase; +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.junit.Assert; import org.junit.Test; -import io.netty.channel.Channel; -import io.netty.channel.local.LocalChannel; - /** * Tests of the main BookKeeper client using networkless comunication */ -public class NetworkLessBookieTest extends BaseTestCase { - - protected ServerConfiguration newServerConfiguration() throws Exception { - return super - .newServerConfiguration() - .setDisableServerSocketBind(true) - .setEnableLocalTransport(true); - } - - DigestType digestType; +public class NetworkLessBookieTest extends BookKeeperClusterTestCase { - public NetworkLessBookieTest(DigestType digestType) { - super(4); - this.digestType=digestType; + public NetworkLessBookieTest() { + super(1); + baseConf.setDisableServerSocketBind(true); + baseConf.setEnableLocalTransport(true); } @Test @@ -60,12 +48,8 @@ public class NetworkLessBookieTest extends BaseTestCase { .setZkServers(zkUtil.getZooKeeperConnectString()) .setZkTimeout(20000); - CountDownLatch l = new CountDownLatch(1); - zkUtil.sleepServer(5, l); - l.await(); - try (BookKeeper bkc = new BookKeeper(conf)) { - try (LedgerHandle h = bkc.createLedger(1,1,digestType, "testPasswd".getBytes())) { + try (LedgerHandle h = bkc.createLedger(1,1, DigestType.CRC32, "testPasswd".getBytes())) { h.addEntry("test".getBytes()); } } diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/AuditorLedgerCheckerTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/AuditorLedgerCheckerTest.java index f2a7316..d62046b 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/AuditorLedgerCheckerTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/AuditorLedgerCheckerTest.java @@ -41,7 +41,6 @@ import java.util.concurrent.CountDownLatch; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; - import org.apache.bookkeeper.client.AsyncCallback.AddCallback; import org.apache.bookkeeper.client.BKException; import org.apache.bookkeeper.client.BookKeeper.DigestType; @@ -57,7 +56,7 @@ import org.apache.bookkeeper.proto.BookkeeperInternalCallbacks; import org.apache.bookkeeper.proto.DataFormats.UnderreplicatedLedgerFormat; import org.apache.bookkeeper.replication.ReplicationException.CompatibilityException; import org.apache.bookkeeper.replication.ReplicationException.UnavailableException; -import org.apache.bookkeeper.test.MultiLedgerManagerTestCase; +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.apache.commons.lang.mutable.MutableInt; import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.WatchedEvent; @@ -72,7 +71,7 @@ import org.slf4j.LoggerFactory; * Tests publishing of under replicated ledgers by the Auditor bookie node when * corresponding bookies identifes as not running */ -public class AuditorLedgerCheckerTest extends MultiLedgerManagerTestCase { +public class AuditorLedgerCheckerTest extends BookKeeperClusterTestCase { // Depending on the taste, select the amount of logging // by decommenting one of the two lines below @@ -95,7 +94,13 @@ public class AuditorLedgerCheckerTest extends MultiLedgerManagerTestCase { private List ledgerList; - public AuditorLedgerCheckerTest(String ledgerManagerFactoryClass) + public AuditorLedgerCheckerTest() + throws IOException, KeeperException, InterruptedException, + CompatibilityException { + this("org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory"); + } + + AuditorLedgerCheckerTest(String ledgerManagerFactoryClass) throws IOException, KeeperException, InterruptedException, CompatibilityException { super(3); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/BookieLedgerIndexTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/BookieLedgerIndexTest.java index 2872397..26a50e4 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/BookieLedgerIndexTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/BookieLedgerIndexTest.java @@ -26,7 +26,6 @@ import java.util.List; import java.util.Map; import java.util.Random; import java.util.Set; - import org.apache.bookkeeper.client.BKException; import org.apache.bookkeeper.client.LedgerHandle; import org.apache.bookkeeper.client.BookKeeper.DigestType; @@ -34,7 +33,7 @@ import org.apache.bookkeeper.meta.LedgerManager; import org.apache.bookkeeper.meta.LedgerManagerFactory; import org.apache.bookkeeper.meta.MSLedgerManagerFactory; import org.apache.bookkeeper.replication.ReplicationException.BKAuditException; -import org.apache.bookkeeper.test.MultiLedgerManagerTestCase; +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.apache.commons.io.FileUtils; import org.apache.zookeeper.KeeperException; import org.junit.After; @@ -48,7 +47,7 @@ import static org.junit.Assert.*; /** * Tests verifies bookie vs ledger mapping generating by the BookieLedgerIndexer */ -public class BookieLedgerIndexTest extends MultiLedgerManagerTestCase { +public class BookieLedgerIndexTest extends BookKeeperClusterTestCase { // Depending on the taste, select the amount of logging // by decommenting one of the two lines below @@ -64,7 +63,12 @@ public class BookieLedgerIndexTest extends MultiLedgerManagerTestCase { private LedgerManagerFactory newLedgerManagerFactory; private LedgerManager ledgerManager; - public BookieLedgerIndexTest(String ledgerManagerFactory) + public BookieLedgerIndexTest() + throws IOException, KeeperException, InterruptedException { + this("org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory"); + } + + BookieLedgerIndexTest(String ledgerManagerFactory) throws IOException, KeeperException, InterruptedException { super(3); LOG.info("Running test case using ledger manager : " diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/TestReplicationWorker.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/TestReplicationWorker.java index 93890fa..445655d 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/TestReplicationWorker.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/TestReplicationWorker.java @@ -29,7 +29,6 @@ import java.util.ArrayList; import java.util.Enumeration; import java.util.Map.Entry; import java.util.Set; - import org.apache.bookkeeper.client.BKException; import org.apache.bookkeeper.client.BookKeeper; import org.apache.bookkeeper.client.ClientUtil; @@ -41,7 +40,7 @@ import org.apache.bookkeeper.meta.LedgerManagerFactory; import org.apache.bookkeeper.meta.LedgerUnderreplicationManager; import org.apache.bookkeeper.net.BookieSocketAddress; import org.apache.bookkeeper.proto.BookieServer; -import org.apache.bookkeeper.test.MultiLedgerManagerTestCase; +import org.apache.bookkeeper.test.BookKeeperClusterTestCase; import org.apache.bookkeeper.util.BookKeeperConstants; import org.apache.bookkeeper.zookeeper.ZooKeeperClient; import org.apache.zookeeper.ZooKeeper; @@ -53,7 +52,7 @@ import org.slf4j.LoggerFactory; * Test the ReplicationWroker, where it has to replicate the fragments from * failed Bookies to given target Bookie. */ -public class TestReplicationWorker extends MultiLedgerManagerTestCase { +public class TestReplicationWorker extends BookKeeperClusterTestCase { private static final byte[] TESTPASSWD = "testpasswd".getBytes(); private static final Logger LOG = LoggerFactory @@ -64,7 +63,11 @@ public class TestReplicationWorker extends MultiLedgerManagerTestCase { private LedgerUnderreplicationManager underReplicationManager; private static byte[] data = "TestReplicationWorker".getBytes(); - public TestReplicationWorker(String ledgerManagerFactory) { + public TestReplicationWorker() { + this("org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory"); + } + + TestReplicationWorker(String ledgerManagerFactory) { super(3); LOG.info("Running test case using ledger manager : " + ledgerManagerFactory); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/AsyncLedgerOpsTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/AsyncLedgerOpsTest.java index a54436e..5f25ce3 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/AsyncLedgerOpsTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/AsyncLedgerOpsTest.java @@ -1,5 +1,3 @@ -package org.apache.bookkeeper.test; - /* * * Licensed to the Apache Software Foundation (ASF) under one @@ -20,6 +18,7 @@ package org.apache.bookkeeper.test; * under the License. * */ +package org.apache.bookkeeper.test; import java.io.IOException; import java.nio.ByteBuffer; @@ -27,7 +26,6 @@ import java.util.ArrayList; import java.util.Enumeration; import java.util.Random; import java.util.Set; - import org.apache.bookkeeper.client.AsyncCallback.AddCallback; import org.apache.bookkeeper.client.LedgerEntry; import org.apache.bookkeeper.client.AsyncCallback.CloseCallback; @@ -50,16 +48,17 @@ import static org.junit.Assert.*; * and three BookKeepers. * */ -public class AsyncLedgerOpsTest extends MultiLedgerManagerMultiDigestTestCase +public class AsyncLedgerOpsTest extends BookKeeperClusterTestCase implements AddCallback, ReadCallback, CreateCallback, CloseCallback, OpenCallback { private final static Logger LOG = LoggerFactory.getLogger(AsyncLedgerOpsTest.class); - DigestType digestType; + private final DigestType digestType; - public AsyncLedgerOpsTest(String ledgerManagerFactory, DigestType digestType) { + public AsyncLedgerOpsTest() { super(3); - this.digestType = digestType; + this.digestType = DigestType.CRC32; + String ledgerManagerFactory = "org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory"; // set ledger manager type baseConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); baseClientConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BaseTestCase.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BaseTestCase.java deleted file mode 100644 index 437f0ae..0000000 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BaseTestCase.java +++ /dev/null @@ -1,49 +0,0 @@ -/* - * - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - * - */ - -package org.apache.bookkeeper.test; - -import java.util.Arrays; -import java.util.Collection; - -import org.apache.bookkeeper.client.BookKeeper.DigestType; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import org.junit.runner.RunWith; -import org.junit.runners.Parameterized; -import org.junit.runners.Parameterized.Parameters; - -@RunWith(Parameterized.class) -public abstract class BaseTestCase extends BookKeeperClusterTestCase { - static final Logger LOG = LoggerFactory.getLogger(BaseTestCase.class); - - public BaseTestCase(int numBookies) { - super(numBookies); - } - - @Parameters - public static Collection configs() { - return Arrays.asList(new Object[][] { {DigestType.MAC }, {DigestType.CRC32}}); - } - -} diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java index 3b8d2a4..83a19f9 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java @@ -21,6 +21,7 @@ package org.apache.bookkeeper.test; +import com.google.common.base.Stopwatch; import java.io.File; import java.io.IOException; import java.net.InetAddress; @@ -53,6 +54,8 @@ import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.ZooKeeper; import org.junit.After; import org.junit.Before; +import org.junit.Rule; +import org.junit.rules.TestName; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -63,21 +66,24 @@ public abstract class BookKeeperClusterTestCase { static final Logger LOG = LoggerFactory.getLogger(BookKeeperClusterTestCase.class); + @Rule + public final TestName runtime = new TestName(); + // ZooKeeper related variables - protected ZooKeeperUtil zkUtil = new ZooKeeperUtil(); + protected final ZooKeeperUtil zkUtil = new ZooKeeperUtil(); protected ZooKeeper zkc; // BookKeeper related variables - protected List tmpDirs = new LinkedList(); - protected List bs = new LinkedList(); - protected List bsConfs = new LinkedList(); + protected final List tmpDirs = new LinkedList(); + protected final List bs = new LinkedList(); + protected final List bsConfs = new LinkedList(); protected int numBookies; protected BookKeeperTestClient bkc; - protected ServerConfiguration baseConf = TestBKConfiguration.newServerConfiguration(); - protected ClientConfiguration baseClientConf = new ClientConfiguration(); + protected final ServerConfiguration baseConf = TestBKConfiguration.newServerConfiguration(); + protected final ClientConfiguration baseClientConf = new ClientConfiguration(); - private Map autoRecoveryProcesses = new HashMap(); + private final Map autoRecoveryProcesses = new HashMap<>(); private boolean isAutoRecoveryEnabled; @@ -93,11 +99,13 @@ public abstract class BookKeeperClusterTestCase { setMetastoreImplClass(baseConf); setMetastoreImplClass(baseClientConf); + Stopwatch sw = Stopwatch.createStarted(); try { // start zookeeper service startZKCluster(); // start bookkeeper service startBKCluster(); + LOG.info("Setup testcase {} in {} ms.", runtime.getMethodName(), sw.elapsed(TimeUnit.MILLISECONDS)); } catch (Exception e) { LOG.error("Error setting up", e); throw e; @@ -106,6 +114,7 @@ public abstract class BookKeeperClusterTestCase { @After public void tearDown() throws Exception { + Stopwatch sw = Stopwatch.createStarted(); LOG.info("TearDown"); // stop bookkeeper service stopBKCluster(); @@ -113,7 +122,7 @@ public abstract class BookKeeperClusterTestCase { stopZKCluster(); // cleanup temp dirs cleanupTempDirs(); - LOG.info("Tearing down test {}", getClass()); + LOG.info("Tearing down test {} in {} ms.", runtime.getMethodName(), sw.elapsed(TimeUnit.MILLISECONDS)); } protected File createTempDir(String prefix, String suffix) throws IOException { @@ -191,7 +200,12 @@ public abstract class BookKeeperClusterTestCase { protected ServerConfiguration newServerConfiguration() throws Exception { File f = createTempDir("bookie", "test"); - int port = PortManager.nextFreePort(); + int port; + if (baseConf.isEnableLocalTransport() || !baseConf.getAllowEphemeralPorts()) { + port = PortManager.nextFreePort(); + } else { + port = 0; + } return newServerConfiguration(port, zkUtil.getZooKeeperConnectString(), f, new File[] { f }); } @@ -492,9 +506,12 @@ public abstract class BookKeeperClusterTestCase { Thread.sleep(1000); // restart them to ensure we can't for (ServerConfiguration conf : bsConfs) { + // ensure the bookie port is loaded correctly + int port = conf.getBookiePort(); if (null != newConf) { conf.loadConf(newConf); } + conf.setBookiePort(port); bs.add(startBookie(conf)); } } @@ -539,12 +556,10 @@ public abstract class BookKeeperClusterTestCase { host = InetAddress.getLocalHost().getCanonicalHostName(); } - while ( (!conf.isForceReadOnlyBookie() && (bkc.getZkHandle().exists( - "/ledgers/available/" + host + ":" + port, false) == null)) || - ( conf.isForceReadOnlyBookie() && ((bkc.getZkHandle().exists( - "/ledgers/available/readonly/" + host + ":" + port, false) == null))) - ) { - Thread.sleep(500); + while (conf.isForceReadOnlyBookie() + && bkc.getZkHandle().exists(conf.getZkLedgersRootPath() +"/available/readonly/" + host + ":" + port, + false) == null) { + Thread.sleep(100); } bkc.readBookiesBlocking(); @@ -580,7 +595,7 @@ public abstract class BookKeeperClusterTestCase { host = InetAddress.getLocalHost().getCanonicalHostName(); } while (bkc.getZkHandle().exists( - "/ledgers/available/" + host + ":" + port, false) == null) { + conf.getZkLedgersRootPath() + "/available/" + host + ":" + port, false) == null) { Thread.sleep(500); } diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java index 2ece28d..e0d7d1f 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java @@ -1,5 +1,3 @@ -package org.apache.bookkeeper.test; - /* * * Licensed to the Apache Software Foundation (ASF) under one @@ -20,13 +18,13 @@ package org.apache.bookkeeper.test; * under the License. * */ +package org.apache.bookkeeper.test; import java.io.IOException; import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.Enumeration; import java.util.Random; - import org.apache.bookkeeper.client.AsyncCallback.AddCallback; import org.apache.bookkeeper.client.BKException; import org.apache.bookkeeper.client.BookKeeperTestClient; @@ -50,7 +48,7 @@ import static org.junit.Assert.*; * */ -public class BookieFailureTest extends MultiLedgerManagerMultiDigestTestCase +public class BookieFailureTest extends BookKeeperClusterTestCase implements AddCallback, ReadCallback { // Depending on the taste, select the amount of logging @@ -68,7 +66,7 @@ public class BookieFailureTest extends MultiLedgerManagerMultiDigestTestCase Random rng; // Random Number Generator ArrayList entries; // generated entries ArrayList entriesSize; - DigestType digestType; + private final DigestType digestType; class SyncObj { int counter; @@ -84,9 +82,10 @@ public class BookieFailureTest extends MultiLedgerManagerMultiDigestTestCase } } - public BookieFailureTest(String ledgerManagerFactory, DigestType digestType) { + public BookieFailureTest() { super(4); - this.digestType = digestType; + this.digestType = DigestType.CRC32; + String ledgerManagerFactory = "org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory"; // set ledger manager baseConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); baseClientConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieJournalRollingTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieJournalRollingTest.java index e6aa423..e79fa33 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieJournalRollingTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieJournalRollingTest.java @@ -1,5 +1,3 @@ -package org.apache.bookkeeper.test; - /* * * Licensed to the Apache Software Foundation (ASF) under one @@ -20,6 +18,8 @@ package org.apache.bookkeeper.test; * under the License. * */ +package org.apache.bookkeeper.test; + import java.io.File; import java.util.Enumeration; import java.util.concurrent.CountDownLatch; @@ -47,11 +47,12 @@ import static org.junit.Assert.*; public class BookieJournalRollingTest extends BookKeeperClusterTestCase { private final static Logger LOG = LoggerFactory.getLogger(BookieJournalRollingTest.class); - DigestType digestType; + private final DigestType digestType; public BookieJournalRollingTest() { super(1); this.digestType = DigestType.CRC32; + this.baseConf.setAllowEphemeralPorts(false); } @Before @@ -217,6 +218,7 @@ public class BookieJournalRollingTest extends BookKeeperClusterTestCase { // set flush interval to a large value ServerConfiguration newConf = TestBKConfiguration.newServerConfiguration(); newConf.setFlushInterval(999999999); + newConf.setAllowEphemeralPorts(false); // restart bookies restartBookies(newConf); @@ -255,6 +257,7 @@ public class BookieJournalRollingTest extends BookKeeperClusterTestCase { // restart bookies with flush interval set to a large value ServerConfiguration newConf = TestBKConfiguration.newServerConfiguration(); newConf.setFlushInterval(999999999); + newConf.setAllowEphemeralPorts(false); // restart bookies restartBookies(newConf); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieReadWriteTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieReadWriteTest.java index 85d657c..6658bc3 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieReadWriteTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieReadWriteTest.java @@ -1,5 +1,3 @@ -package org.apache.bookkeeper.test; - /* * * Licensed to the Apache Software Foundation (ASF) under one @@ -20,6 +18,7 @@ package org.apache.bookkeeper.test; * under the License. * */ +package org.apache.bookkeeper.test; import java.io.File; import java.io.IOException; @@ -31,7 +30,6 @@ import java.util.Random; import java.util.Arrays; import java.util.concurrent.CountDownLatch; import java.util.concurrent.atomic.AtomicInteger; - import org.apache.bookkeeper.client.AsyncCallback.AddCallback; import org.apache.bookkeeper.client.AsyncCallback.ReadCallback; import org.apache.bookkeeper.client.AsyncCallback.ReadLastConfirmedCallback; @@ -57,8 +55,7 @@ import static org.junit.Assert.*; * and three BookKeepers. * */ - -public class BookieReadWriteTest extends MultiLedgerManagerMultiDigestTestCase +public class BookieReadWriteTest extends BookKeeperClusterTestCase implements AddCallback, ReadCallback, ReadLastConfirmedCallback { // Depending on the taste, select the amount of logging @@ -77,11 +74,12 @@ public class BookieReadWriteTest extends MultiLedgerManagerMultiDigestTestCase ArrayList entries; // generated entries ArrayList entriesSize; - DigestType digestType; + private final DigestType digestType; - public BookieReadWriteTest(String ledgerManagerFactory, DigestType digestType) { + public BookieReadWriteTest() { super(3); - this.digestType = digestType; + this.digestType = DigestType.CRC32; + String ledgerManagerFactory = "org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory"; // set ledger manager baseConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); baseClientConf.setLedgerManagerFactoryClassName(ledgerManagerFactory); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/CloseTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/CloseTest.java index 98c5f10..d090c4a 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/CloseTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/CloseTest.java @@ -1,5 +1,3 @@ -package org.apache.bookkeeper.test; - /* * * Licensed to the Apache Software Foundation (ASF) under one @@ -20,26 +18,24 @@ package org.apache.bookkeeper.test; * under the License. * */ +package org.apache.bookkeeper.test; import org.junit.*; import org.apache.bookkeeper.client.LedgerHandle; import org.apache.bookkeeper.client.BookKeeper.DigestType; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; /** * This unit test tests closing ledgers sequentially. It creates 4 ledgers, then * write 1000 entries to each ledger and close it. * */ +public class CloseTest extends BookKeeperClusterTestCase { -public class CloseTest extends BaseTestCase { - private final static Logger LOG = LoggerFactory.getLogger(CloseTest.class); - DigestType digestType; + private final DigestType digestType; - public CloseTest(DigestType digestType) { + public CloseTest() { super(3); - this.digestType = digestType; + this.digestType = DigestType.CRC32; } @Test(timeout=60000) diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/ConditionalSetTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/ConditionalSetTest.java index 6655eab..4ffaada 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/ConditionalSetTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/ConditionalSetTest.java @@ -21,7 +21,6 @@ package org.apache.bookkeeper.test; import java.io.IOException; - import org.apache.bookkeeper.client.BookKeeperTestClient; import org.apache.bookkeeper.client.BKException; import org.apache.bookkeeper.client.BookKeeper; @@ -30,7 +29,6 @@ import org.apache.bookkeeper.client.BookKeeper.DigestType; import org.apache.zookeeper.KeeperException; import org.junit.Before; import org.junit.Test; - import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -39,16 +37,16 @@ import static org.junit.Assert.*; /** * Tests conditional set of the ledger metadata znode. */ -public class ConditionalSetTest extends BaseTestCase { +public class ConditionalSetTest extends BookKeeperClusterTestCase { private final static Logger LOG = LoggerFactory.getLogger(ConditionalSetTest.class); byte[] entry; - DigestType digestType; + private final DigestType digestType; BookKeeper bkcReader; - public ConditionalSetTest(DigestType digestType) { + public ConditionalSetTest() { super(3); - this.digestType = digestType; + this.digestType = DigestType.CRC32; } @Override diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerDeleteTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerDeleteTest.java index 0c1a187..ed2b1eb 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerDeleteTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerDeleteTest.java @@ -1,5 +1,3 @@ -package org.apache.bookkeeper.test; - /* * * Licensed to the Apache Software Foundation (ASF) under one @@ -20,11 +18,11 @@ package org.apache.bookkeeper.test; * under the License. * */ +package org.apache.bookkeeper.test; import java.io.File; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.CountDownLatch; - import org.apache.bookkeeper.bookie.InterleavedLedgerStorage; import org.apache.bookkeeper.client.BKException; import org.apache.bookkeeper.client.LedgerHandle; @@ -42,11 +40,15 @@ import static org.junit.Assert.*; * This class tests the ledger delete functionality both from the BookKeeper * client and the server side. */ -public class LedgerDeleteTest extends MultiLedgerManagerTestCase { +public class LedgerDeleteTest extends BookKeeperClusterTestCase { private final static Logger LOG = LoggerFactory.getLogger(LedgerDeleteTest.class); DigestType digestType; - public LedgerDeleteTest(String ledgerManagerFactory) { + public LedgerDeleteTest() { + this("org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory"); + } + + LedgerDeleteTest(String ledgerManagerFactory) { super(1); LOG.info("Running test case using ledger manager : " + ledgerManagerFactory); this.digestType = DigestType.CRC32; diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LocalBookiesRegistryTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LocalBookiesRegistryTest.java index 25c5ff4..03d1efe 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LocalBookiesRegistryTest.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LocalBookiesRegistryTest.java @@ -21,32 +21,22 @@ package org.apache.bookkeeper.test; -import org.apache.bookkeeper.client.BookKeeper; -import org.apache.bookkeeper.conf.ServerConfiguration; -import org.apache.bookkeeper.proto.BookieServer; -import org.apache.bookkeeper.proto.LocalBookiesRegistry; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; + +import org.apache.bookkeeper.proto.BookieServer; +import org.apache.bookkeeper.proto.LocalBookiesRegistry; import org.junit.Test; /** * Test the correctness and the availability outside of its package of LocalBookiesRegistryTest */ -public class LocalBookiesRegistryTest extends BaseTestCase { - - @Override - protected ServerConfiguration newServerConfiguration() throws Exception { - return super - .newServerConfiguration() - .setDisableServerSocketBind(true) - .setEnableLocalTransport(true); - } - - BookKeeper.DigestType digestType; +public class LocalBookiesRegistryTest extends BookKeeperClusterTestCase { - public LocalBookiesRegistryTest(BookKeeper.DigestType digestType) { + public LocalBookiesRegistryTest() { super(3); - this.digestType = digestType; + baseConf.setDisableServerSocketBind(true); + baseConf.setEnableLocalTransport(true); } @Test diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerMultiDigestTestCase.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerMultiDigestTestCase.java deleted file mode 100644 index 357bd34..0000000 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerMultiDigestTestCase.java +++ /dev/null @@ -1,66 +0,0 @@ -/* - * - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - * - */ - -package org.apache.bookkeeper.test; - -import java.util.ArrayList; -import java.util.Collection; - -import org.apache.bookkeeper.client.BookKeeper.DigestType; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import org.junit.runner.RunWith; -import org.junit.runners.Parameterized; -import org.junit.runners.Parameterized.Parameters; - -/** - * Running test case using different ledger managers. - */ -@RunWith(Parameterized.class) -public abstract class MultiLedgerManagerMultiDigestTestCase extends BookKeeperClusterTestCase { - static final Logger LOG = LoggerFactory.getLogger(MultiLedgerManagerMultiDigestTestCase.class); - - public MultiLedgerManagerMultiDigestTestCase(int numBookies) { - super(numBookies); - } - - @Parameters - public static Collection configs() { - String[] ledgerManagers = { - "org.apache.bookkeeper.meta.FlatLedgerManagerFactory", - "org.apache.bookkeeper.meta.LegacyHierarchicalLedgerManagerFactory", - "org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory", - "org.apache.bookkeeper.meta.LongHierarchicalLedgerManagerFactory", - "org.apache.bookkeeper.meta.MSLedgerManagerFactory", - }; - ArrayList cfgs = new ArrayList(ledgerManagers.length); - DigestType[] digestTypes = new DigestType[] { DigestType.MAC, DigestType.CRC32 }; - for (String lm : ledgerManagers) { - for (DigestType type : digestTypes) { - cfgs.add(new Object[] { lm, type }); - } - } - return cfgs; - } - -} diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerTestCase.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerTestCase.java deleted file mode 100644 index cb640b1..0000000 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerTestCase.java +++ /dev/null @@ -1,56 +0,0 @@ -package org.apache.bookkeeper.test; - -import java.util.ArrayList; -import java.util.Collection; - -import org.junit.runner.RunWith; -import org.junit.runners.Parameterized; -import org.junit.runners.Parameterized.Parameters; - -/* - * - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - * - */ - -/** - * Test Case run over different ledger manager. - */ -@RunWith(Parameterized.class) -public abstract class MultiLedgerManagerTestCase extends BookKeeperClusterTestCase { - - public MultiLedgerManagerTestCase(int numBookies) { - super(numBookies); - } - - @Parameters - public static Collection configs() { - String[] ledgerManagers = new String[] { - "org.apache.bookkeeper.meta.FlatLedgerManagerFactory", - "org.apache.bookkeeper.meta.LegacyHierarchicalLedgerManagerFactory", - "org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory", - "org.apache.bookkeeper.meta.LongHierarchicalLedgerManagerFactory", - "org.apache.bookkeeper.meta.MSLedgerManagerFactory", - }; - ArrayList cfgs = new ArrayList(ledgerManagers.length); - for (String lm : ledgerManagers) { - cfgs.add(new Object[] { lm }); - } - return cfgs; - } -} diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/TestBackwardCompat.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/TestBackwardCompat.java index b5d01b3..951d4d8 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/TestBackwardCompat.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/TestBackwardCompat.java @@ -31,11 +31,9 @@ import java.util.Arrays; import java.util.Enumeration; import java.util.ArrayList; import java.util.List; - import org.apache.commons.io.FileUtils; import org.apache.bookkeeper.bookie.Bookie; import org.apache.bookkeeper.bookie.BookieException; -import org.apache.bookkeeper.bookie.FileSystemUpgrade; import org.apache.bookkeeper.client.BookKeeperAdmin; import org.apache.bookkeeper.conf.AbstractConfiguration; import org.apache.bookkeeper.conf.ClientConfiguration; @@ -84,103 +82,6 @@ public class TestBackwardCompat { } /** - * Version 4.0.0 classes - */ - static class Server400 { - org.apache.bk_v4_0_0.bookkeeper.conf.ServerConfiguration conf; - org.apache.bk_v4_0_0.bookkeeper.proto.BookieServer server = null; - - Server400(File journalDir, File ledgerDir, int port) throws Exception { - conf = new org.apache.bk_v4_0_0.bookkeeper.conf.ServerConfiguration(); - conf.setBookiePort(port); - conf.setZkServers(zkUtil.getZooKeeperConnectString()); - conf.setJournalDirName(journalDir.getPath()); - conf.setLedgerDirNames(new String[] { ledgerDir.getPath() }); - } - - void start() throws Exception { - server = new org.apache.bk_v4_0_0.bookkeeper.proto.BookieServer(conf); - server.start(); - waitUp(conf.getBookiePort()); - } - - org.apache.bk_v4_0_0.bookkeeper.conf.ServerConfiguration getConf() { - return conf; - } - - void stop() throws Exception { - if (server != null) { - server.shutdown(); - } - } - } - - static class Ledger400 { - org.apache.bk_v4_0_0.bookkeeper.client.BookKeeper bk; - org.apache.bk_v4_0_0.bookkeeper.client.LedgerHandle lh; - - private Ledger400(org.apache.bk_v4_0_0.bookkeeper.client.BookKeeper bk, - org.apache.bk_v4_0_0.bookkeeper.client.LedgerHandle lh) { - this.bk = bk; - this.lh = lh; - } - - static Ledger400 newLedger() throws Exception { - org.apache.bk_v4_0_0.bookkeeper.client.BookKeeper newbk - = new org.apache.bk_v4_0_0.bookkeeper.client.BookKeeper(zkUtil.getZooKeeperConnectString()); - org.apache.bk_v4_0_0.bookkeeper.client.LedgerHandle newlh - = newbk.createLedger(1, 1, - org.apache.bk_v4_0_0.bookkeeper.client.BookKeeper.DigestType.CRC32, - "foobar".getBytes()); - return new Ledger400(newbk, newlh); - } - - static Ledger400 openLedger(long id) throws Exception { - org.apache.bk_v4_0_0.bookkeeper.client.BookKeeper newbk - = new org.apache.bk_v4_0_0.bookkeeper.client.BookKeeper(zkUtil.getZooKeeperConnectString()); - org.apache.bk_v4_0_0.bookkeeper.client.LedgerHandle newlh - = newbk.openLedger(id, - org.apache.bk_v4_0_0.bookkeeper.client.BookKeeper.DigestType.CRC32, - "foobar".getBytes()); - return new Ledger400(newbk, newlh); - } - - long getId() { - return lh.getId(); - } - - void write100() throws Exception { - for (int i = 0; i < 100; i++) { - lh.addEntry(ENTRY_DATA); - } - } - - long readAll() throws Exception { - long count = 0; - Enumeration entries - = lh.readEntries(0, lh.getLastAddConfirmed()); - while (entries.hasMoreElements()) { - assertTrue("entry data doesn't match", - Arrays.equals(entries.nextElement().getEntry(), ENTRY_DATA)); - count++; - } - return count; - } - - void close() throws Exception { - try { - if (lh != null) { - lh.close(); - } - } finally { - if (bk != null) { - bk.close(); - } - } - } - } - - /** * Version 4.1.0 classes */ static class Server410 { @@ -290,6 +191,7 @@ public class TestBackwardCompat { conf.setZkServers(zkUtil.getZooKeeperConnectString()); conf.setJournalDirName(journalDir.getPath()); conf.setLedgerDirNames(new String[] { ledgerDir.getPath() }); + conf.setDiskUsageThreshold(0.999f); } void start() throws Exception { @@ -384,6 +286,7 @@ public class TestBackwardCompat { ServerCurrent(File journalDir, File ledgerDir, int port, boolean useHostNameAsBookieID) throws Exception { conf = TestBKConfiguration.newServerConfiguration(); + conf.setAllowEphemeralPorts(false); conf.setBookiePort(port); conf.setZkServers(zkUtil.getZooKeeperConnectString()); conf.setJournalDirName(journalDir.getPath()); @@ -535,91 +438,6 @@ public class TestBackwardCompat { } /** - * Test compatability between version 4.0.0 and the current version. - * Incompatabilities are: - * - Current client will not be able to talk to 4.0.0 server. - * - 4.0.0 client will not be able to fence ledgers on current server. - * - Current server won't start with 4.0.0 server directories without upgrade. - */ - @Test(timeout=60000) - public void testCompat400() throws Exception { - File journalDir = createTempDir("bookie", "journal"); - File ledgerDir = createTempDir("bookie", "ledger"); - - int port = PortManager.nextFreePort(); - // start server, upgrade - Server400 s400 = new Server400(journalDir, ledgerDir, port); - s400.start(); - - Ledger400 l400 = Ledger400.newLedger(); - l400.write100(); - long oldLedgerId = l400.getId(); - l400.close(); - - // Check that current client isn't able to write to old server - LedgerCurrent lcur = LedgerCurrent.newLedger(); - try { - lcur.write100(); - fail("Current shouldn't be able to write to 4.0.0 server"); - } catch (Exception e) { - } - lcur.close(); - - s400.stop(); - - // Start the current server, will require a filesystem upgrade - ServerCurrent scur = new ServerCurrent(journalDir, ledgerDir, port, false); - try { - scur.start(); - fail("Shouldn't be able to start without directory upgrade"); - } catch (Exception e) { - } - FileSystemUpgrade.upgrade(scur.getConf()); - - scur.start(); - - // check that old client can read its old ledgers on new server - l400 = Ledger400.openLedger(oldLedgerId); - assertEquals(100, l400.readAll()); - l400.close(); - - // check that old client can create ledgers on new server - l400 = Ledger400.newLedger(); - l400.write100(); - l400.close(); - - // check that current client can read old ledger - lcur = LedgerCurrent.openLedger(oldLedgerId); - assertEquals(100, lcur.readAll()); - lcur.close(); - - // check that old client can read current client's ledgers - lcur = LedgerCurrent.openLedger(oldLedgerId); - assertEquals(100, lcur.readAll()); - lcur.close(); - - // check that old client can not fence a current client - // due to lack of password - lcur = LedgerCurrent.newLedger(); - lcur.write100(); - long fenceLedgerId = lcur.getId(); - try { - l400 = Ledger400.openLedger(fenceLedgerId); - fail("Shouldn't be able to open ledger"); - } catch (Exception e) { - // correct behaviour - } - lcur.write100(); - lcur.close(); - - lcur = LedgerCurrent.openLedger(fenceLedgerId); - assertEquals(200, lcur.readAll()); - lcur.close(); - - scur.stop(); - } - - /** * Test compatability between version 4.1.0 and the current version. * - A 4.1.0 client is not able to open a ledger created by the current * version due to a change in the ledger metadata format. diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/ZooKeeperUtil.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/ZooKeeperUtil.java index 5fc95cb..a6b9b87 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/ZooKeeperUtil.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/ZooKeeperUtil.java @@ -25,25 +25,22 @@ import java.io.File; import java.io.IOException; import java.net.InetAddress; import java.net.InetSocketAddress; - +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; import org.apache.bookkeeper.util.IOUtils; import org.apache.bookkeeper.zookeeper.ZooKeeperClient; import org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase; import org.apache.commons.io.FileUtils; - -import java.util.concurrent.CountDownLatch; - -import org.apache.zookeeper.ZooKeeper; import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.Transaction; import org.apache.zookeeper.ZooDefs.Ids; - +import org.apache.zookeeper.ZooKeeper; import org.apache.zookeeper.server.NIOServerCnxnFactory; import org.apache.zookeeper.server.ZooKeeperServer; import org.apache.zookeeper.test.ClientBase; - import org.slf4j.Logger; import org.slf4j.LoggerFactory; - import static org.junit.Assert.*; public class ZooKeeperUtil { @@ -89,9 +86,15 @@ public class ZooKeeperUtil { // start the server and client. restartServer(); - // initialize the zk client with values - zkc.create("/ledgers", new byte[0], Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); - zkc.create("/ledgers/available", new byte[0], Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); + // create default bk ensemble + createBKEnsemble("/ledgers"); + } + + public void createBKEnsemble(String ledgersPath) throws KeeperException, InterruptedException { + Transaction txn = zkc.transaction(); + txn.create(ledgersPath, new byte[0], Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); + txn.create(ledgersPath + "/available", new byte[0], Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); + txn.commit(); } public void restartServer() throws Exception { @@ -113,7 +116,9 @@ public class ZooKeeperUtil { .build(); } - public void sleepServer(final int seconds, final CountDownLatch l) + public void sleepServer(final int time, + final TimeUnit timeUnit, + final CountDownLatch l) throws InterruptedException, IOException { Thread[] allthreads = new Thread[Thread.activeCount()]; Thread.enumerate(allthreads); @@ -125,7 +130,7 @@ public class ZooKeeperUtil { try { t.suspend(); l.countDown(); - Thread.sleep(seconds*1000); + timeUnit.sleep(time); t.resume(); } catch (Exception e) { LOG.error("Error suspending thread", e); diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/util/TestDiskChecker.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/util/TestDiskChecker.java index 2f4bbc5..b09747c 100644 --- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/util/TestDiskChecker.java +++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/util/TestDiskChecker.java @@ -24,7 +24,6 @@ import java.io.FileOutputStream; import java.io.IOException; import java.util.ArrayList; import java.util.List; - import org.apache.bookkeeper.util.DiskChecker.DiskErrorException; import org.apache.bookkeeper.util.DiskChecker.DiskOutOfSpaceException; import org.apache.bookkeeper.util.DiskChecker.DiskWarnThresholdException; @@ -42,10 +41,11 @@ public class TestDiskChecker { DiskChecker diskChecker; final List tempDirs = new ArrayList(); + private static final float THRESHOLD = 0.99f; @Before public void setup() throws IOException { - diskChecker = new DiskChecker(0.95f, 0.95f); + diskChecker = new DiskChecker(THRESHOLD, THRESHOLD); // Create at least one file so that target disk will never be empty File placeHolderDir = IOUtils.createTempDir("DiskCheck", "test-placeholder"); @@ -78,7 +78,7 @@ public class TestDiskChecker { File file = createTempDir("DiskCheck", "test"); long usableSpace = file.getUsableSpace(); long totalSpace = file.getTotalSpace(); - float threshold = minMaxThreshold((1f - ((float) usableSpace / (float) totalSpace)) - 0.05f); + float threshold = minMaxThreshold((1f - ((float) usableSpace / (float) totalSpace)) - (1.0f - THRESHOLD)); diskChecker.setDiskSpaceThreshold(threshold, threshold); diskChecker.checkDiskFull(file); -- To stop receiving notification emails like this one, please contact ['"commits@bookkeeper.apache.org" '].