ratis-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From msi...@apache.org
Subject incubator-ratis git commit: RATIS-456. Add a load generator for Filestore example. Contributed by Mukul Kumar Singh.
Date Thu, 20 Dec 2018 07:00:12 GMT
Repository: incubator-ratis
Updated Branches:
  refs/heads/master c038d2e21 -> 09cc126d7


RATIS-456. Add a load generator for Filestore example. Contributed by Mukul Kumar Singh.


Project: http://git-wip-us.apache.org/repos/asf/incubator-ratis/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-ratis/commit/09cc126d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-ratis/tree/09cc126d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-ratis/diff/09cc126d

Branch: refs/heads/master
Commit: 09cc126d7364fdfc23cac4d82af1fc84310c3884
Parents: c038d2e
Author: Mukul Kumar Singh <msingh@apache.org>
Authored: Thu Dec 20 12:27:59 2018 +0530
Committer: Mukul Kumar Singh <msingh@apache.org>
Committed: Thu Dec 20 12:27:59 2018 +0530

----------------------------------------------------------------------
 README.md                                       | 40 ++++++++
 ratis-examples/pom.xml                          |  7 +-
 ratis-examples/src/main/bin/client.sh           |  2 +-
 ratis-examples/src/main/bin/server.sh           |  6 +-
 .../ratis/examples/arithmetic/Runner.java       | 79 ----------------
 .../examples/arithmetic/cli/Arithmetic.java     | 37 ++++++++
 .../ratis/examples/arithmetic/cli/Client.java   |  1 +
 .../ratis/examples/arithmetic/cli/Server.java   |  1 +
 .../examples/arithmetic/cli/SubCommandBase.java | 53 -----------
 .../apache/ratis/examples/common/Runner.java    | 87 ++++++++++++++++++
 .../ratis/examples/common/SubCommandBase.java   | 53 +++++++++++
 .../examples/filestore/FileStoreClient.java     |  4 +
 .../ratis/examples/filestore/cli/Client.java    | 83 +++++++++++++++++
 .../ratis/examples/filestore/cli/FileStore.java | 36 ++++++++
 .../ratis/examples/filestore/cli/LoadGen.java   | 93 +++++++++++++++++++
 .../ratis/examples/filestore/cli/Server.java    | 96 ++++++++++++++++++++
 16 files changed, 539 insertions(+), 139 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 650e5f4..03aa89b 100644
--- a/README.md
+++ b/README.md
@@ -20,6 +20,46 @@ The paper introduces Raft and states its motivations in following words:
 
   Ratis aims to make raft available as a java library that can be used by any system that
needs to use a replicated log. It provides pluggability for state machine implementations
to manage replicated states. It also provides pluggability for Raft log, and rpc implementations
to make it easy for integration with other projects. Another important goal is to support
high throughput data ingest so that it can be used for more general data replication use cases.
 
+# Usage
+
+Compile the repository using `mvn clean package -DskipTests`
+
+## FileStore
+
+### Server
+To spawn the FileStoreStateMachineServer, use `bin/server.sh filestore server --id <selfid>
--storage <storage_dir> --peers <id:ip_address,...>`
+
+selfid is the id of the instance being spwaned, this should be one of the ids in the peer
list.
+
+For example `ratis-examples/src/main/bin/server.sh filestore server --id n0 --storage /tmp/data
--peers n0:172.26.32.224:6000,n1:172.26.32.225:6001,n2:172.26.32.226:6002`
+
+### Client
+
+To spawn the FileStoreStateMachine client, use `bin/client.sh filestore loadgen --value <file_size>
--files <num_files> --peers <id:ip_address,...>`
+
+Where, file_size is the size of the file to be generated in bytes, num_files is the number
of files to be generated.
+
+For example `ratis-examples/src/main/bin/client.sh filestore loadgen --value 1048576 --files
1000 --peers n0:172.26.32.224:6000,n1:172.26.32.225:6001,n2:172.26.32.226:6002`
+
+## Arithmetic
+
+### Server
+To spawn the ArithmeticStateMachineServer, use `bin/server.sh arithmetic server --id <selfid>
--storage <storage_dir> --peers <id:ip_address,...>`
+
+selfid is the id of the instance being spwaned, this should be one of the ids in the peer
list.
+
+For example `ratis-examples/src/main/bin/server.sh arithmetic server --id n0 --storage /tmp/data
--peers n0:172.26.32.224:6000,n1:172.26.32.225:6001,n2:172.26.32.226:6002`
+
+### Client
+
+To spawn the ArithmeticStateMachine client, use `bin/client.sh arithmetic get --name b --peers
<id:ip_address,...>`
+
+Where, b is the name of the variable.
+
+For example `ratis-examples/src/main/bin/client.sh arithmetic get --name b --peers n0:172.26.32.224:6000,n1:172.26.32.225:6001,n2:172.26.32.226:6002`
+
+PS: the peer is a id, ipaddress pair seperated by ':', for eg. n0:172.26.32.224:6000
+
 
 # Reference
 [1] _Diego Ongaro and John Ousterhout. 2014. In search of an understandable consensus algorithm.
In Proceedings of the 2014 USENIX conference on USENIX Annual Technical Conference (USENIX
ATC'14), Garth Gibson and Nickolai Zeldovich (Eds.). USENIX Association, Berkeley, CA, USA,
305-320._

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/pom.xml
----------------------------------------------------------------------
diff --git a/ratis-examples/pom.xml b/ratis-examples/pom.xml
index ae88124..19c26dd 100644
--- a/ratis-examples/pom.xml
+++ b/ratis-examples/pom.xml
@@ -54,6 +54,11 @@
       <groupId>org.apache.ratis</groupId>
     </dependency>
     <dependency>
+      <groupId>org.apache.commons</groupId>
+      <artifactId>commons-lang3</artifactId>
+      <version>3.5</version>
+    </dependency>
+    <dependency>
       <artifactId>ratis-server</artifactId>
       <groupId>org.apache.ratis</groupId>
       <scope>test</scope>
@@ -145,7 +150,7 @@
               <transformers>
                 <transformer
                         implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
-                  <mainClass>org.apache.ratis.examples.arithmetic.Runner
+                  <mainClass>org.apache.ratis.examples.common.Runner
                   </mainClass>
                 </transformer>
               </transformers>

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/bin/client.sh
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/bin/client.sh b/ratis-examples/src/main/bin/client.sh
index 60946a3..edcaaea 100755
--- a/ratis-examples/src/main/bin/client.sh
+++ b/ratis-examples/src/main/bin/client.sh
@@ -19,5 +19,5 @@ source $DIR/common.sh
 
 subcommand=$1
 shift
-java -jar $ARTIFACT $subcommand --peers n0:localhost:6000,n1:localhost:6001,n2:localhost:6002
"$@"
+java -jar $ARTIFACT $subcommand "$@"
 

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/bin/server.sh
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/bin/server.sh b/ratis-examples/src/main/bin/server.sh
index 564daa1..64b99f1 100755
--- a/ratis-examples/src/main/bin/server.sh
+++ b/ratis-examples/src/main/bin/server.sh
@@ -16,9 +16,5 @@
 
 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 source $DIR/common.sh
-if [ -z "$1" ]; then
-   echo "Usage: server.sh <nodeid>"
-   exit -1
-fi
-java -jar $ARTIFACT server --storage /tmp/ratis-arithmentic-$1 --id $1 --peers n0:localhost:6000,n1:localhost:6001,n2:localhost:6002
+java -jar $ARTIFACT "$@"
 

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/Runner.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/Runner.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/Runner.java
deleted file mode 100644
index ab189a8..0000000
--- a/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/Runner.java
+++ /dev/null
@@ -1,79 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.ratis.examples.arithmetic;
-
-import com.beust.jcommander.JCommander;
-import com.beust.jcommander.ParameterException;
-import org.apache.log4j.Level;
-import org.apache.ratis.client.RaftClient;
-import org.apache.ratis.examples.arithmetic.cli.Assign;
-import org.apache.ratis.examples.arithmetic.cli.Get;
-import org.apache.ratis.examples.arithmetic.cli.SubCommandBase;
-import org.apache.ratis.examples.arithmetic.cli.Server;
-import org.apache.ratis.server.impl.RaftServerImpl;
-import org.apache.ratis.util.LogUtils;
-
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Optional;
-
-/**
- * Standalone arithmetic raft server.
- */
-public class Runner {
-
-  private static List<SubCommandBase> commands = new ArrayList<>();
-
-  static {
-    LogUtils.setLogLevel(RaftServerImpl.LOG, Level.DEBUG);
-    LogUtils.setLogLevel(RaftClient.LOG, Level.DEBUG);
-  }
-
-  public static void main(String[] args) throws Exception {
-    initializeCommands();
-    Runner runner = new Runner();
-    Server server = new Server();
-
-    JCommander.Builder builder = JCommander.newBuilder().addObject(runner);
-    commands.forEach(command -> builder
-        .addCommand(command.getClass().getSimpleName().toLowerCase(), command));
-    JCommander jc = builder.build();
-    try {
-      jc.parse(args);
-      Optional<SubCommandBase> selectedCommand = commands.stream().filter(
-          command -> command.getClass().getSimpleName().toLowerCase()
-              .equals(jc.getParsedCommand())).findFirst();
-      if (selectedCommand.isPresent()) {
-        selectedCommand.get().run();
-      } else {
-        jc.usage();
-      }
-    } catch (ParameterException exception) {
-      System.err.println("Wrong parameters: " + exception.getMessage());
-      jc.usage();
-    }
-
-  }
-
-  private static void initializeCommands() {
-    commands.add(new Server());
-    commands.add(new Assign());
-    commands.add(new Get());
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Arithmetic.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Arithmetic.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Arithmetic.java
new file mode 100644
index 0000000..fc35348
--- /dev/null
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Arithmetic.java
@@ -0,0 +1,37 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ratis.examples.arithmetic.cli;
+
+import org.apache.ratis.examples.common.SubCommandBase;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This class enumerates all the commands enqueued by Arithmetic state machine.
+ */
+public class Arithmetic {
+  public static List<SubCommandBase> getSubCommands() {
+    List<SubCommandBase> commands = new ArrayList<>();
+    commands.add(new Server());
+    commands.add(new Assign());
+    commands.add(new Get());
+    return commands;
+  }
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Client.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Client.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Client.java
index f7647bd..5efbcb8 100644
--- a/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Client.java
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Client.java
@@ -20,6 +20,7 @@ package org.apache.ratis.examples.arithmetic.cli;
 import org.apache.ratis.client.RaftClient;
 import org.apache.ratis.conf.Parameters;
 import org.apache.ratis.conf.RaftProperties;
+import org.apache.ratis.examples.common.SubCommandBase;
 import org.apache.ratis.grpc.GrpcFactory;
 import org.apache.ratis.protocol.ClientId;
 import org.apache.ratis.protocol.RaftGroup;

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Server.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Server.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Server.java
index 1772073..5089bd7 100644
--- a/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Server.java
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/Server.java
@@ -21,6 +21,7 @@ import com.beust.jcommander.Parameter;
 import com.beust.jcommander.Parameters;
 import org.apache.ratis.conf.RaftProperties;
 import org.apache.ratis.examples.arithmetic.ArithmeticStateMachine;
+import org.apache.ratis.examples.common.SubCommandBase;
 import org.apache.ratis.grpc.GrpcConfigKeys;
 import org.apache.ratis.protocol.RaftGroup;
 import org.apache.ratis.protocol.RaftGroupId;

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/SubCommandBase.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/SubCommandBase.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/SubCommandBase.java
deleted file mode 100644
index 0c982e4..0000000
--- a/ratis-examples/src/main/java/org/apache/ratis/examples/arithmetic/cli/SubCommandBase.java
+++ /dev/null
@@ -1,53 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.ratis.examples.arithmetic.cli;
-
-import com.beust.jcommander.Parameter;
-import org.apache.ratis.protocol.RaftPeer;
-import org.apache.ratis.protocol.RaftPeerId;
-
-import java.util.stream.Stream;
-
-/**
- * Base subcommand class which includes the basic raft properties.
- */
-public abstract class SubCommandBase {
-
-  @Parameter(names = {"--raftGroup",
-      "-g"}, description = "Raft group identifier")
-  protected String raftGroupId = "demoRaftGroup123";
-
-  @Parameter(names = {"--peers", "-r"}, description =
-      "Raft peers (format: name:host:port,"
-          + "name:host:port)", required = true)
-  protected String peers;
-
-  public static RaftPeer[] parsePeers(String peers) {
-    return Stream.of(peers.split(",")).map(address -> {
-      String[] addressParts = address.split(":");
-      return new RaftPeer(RaftPeerId.valueOf(addressParts[0]),
-          addressParts[1] + ":" + addressParts[2]);
-    }).toArray(RaftPeer[]::new);
-  }
-
-  public RaftPeer[] getPeers() {
-    return parsePeers(peers);
-  }
-
-  public abstract void run() throws Exception;
-}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/common/Runner.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/common/Runner.java b/ratis-examples/src/main/java/org/apache/ratis/examples/common/Runner.java
new file mode 100644
index 0000000..944ce6b
--- /dev/null
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/common/Runner.java
@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.examples.common;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.ParameterException;
+import org.apache.log4j.Level;
+import org.apache.ratis.client.RaftClient;
+import org.apache.ratis.examples.arithmetic.cli.Arithmetic;
+import org.apache.ratis.examples.filestore.cli.FileStore;
+import org.apache.ratis.server.impl.RaftServerImpl;
+import org.apache.ratis.util.LogUtils;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * Standalone arithmetic raft server.
+ */
+public class Runner {
+
+  static {
+    LogUtils.setLogLevel(RaftServerImpl.LOG, Level.DEBUG);
+    LogUtils.setLogLevel(RaftClient.LOG, Level.DEBUG);
+  }
+
+  public static void main(String[] args) throws Exception {
+    if (args.length == 0) {
+      System.err.println("No command type specified: ");
+      return;
+    }
+    List<SubCommandBase> commands = initializeCommands(args[0]);
+    Runner runner = new Runner();
+
+    if (commands == null) {
+      System.err.println("Wrong command type: " + args[0]);
+      return;
+    }
+    String[] newArgs = new String[args.length - 1];
+    System.arraycopy(args, 1, newArgs, 0, args.length - 1);
+
+    JCommander.Builder builder = JCommander.newBuilder().addObject(runner);
+    commands.forEach(command -> builder
+        .addCommand(command.getClass().getSimpleName().toLowerCase(), command));
+    JCommander jc = builder.build();
+    try {
+      jc.parse(newArgs);
+      Optional<SubCommandBase> selectedCommand = commands.stream().filter(
+          command -> command.getClass().getSimpleName().toLowerCase()
+              .equals(jc.getParsedCommand())).findFirst();
+      if (selectedCommand.isPresent()) {
+        selectedCommand.get().run();
+      } else {
+        jc.usage();
+      }
+    } catch (ParameterException exception) {
+      System.err.println("Wrong parameters: " + exception.getMessage());
+      jc.usage();
+    }
+
+  }
+
+  private static List<SubCommandBase> initializeCommands(String command) {
+    if (command.equals(FileStore.class.getSimpleName().toLowerCase())) {
+      return FileStore.getSubCommands();
+    } else if (command.equals(Arithmetic.class.getSimpleName().toLowerCase())) {
+      return Arithmetic.getSubCommands();
+    }
+    return null;
+  }
+
+}

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/common/SubCommandBase.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/common/SubCommandBase.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/common/SubCommandBase.java
new file mode 100644
index 0000000..7f3d402
--- /dev/null
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/common/SubCommandBase.java
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.examples.common;
+
+import com.beust.jcommander.Parameter;
+import org.apache.ratis.protocol.RaftPeer;
+import org.apache.ratis.protocol.RaftPeerId;
+
+import java.util.stream.Stream;
+
+/**
+ * Base subcommand class which includes the basic raft properties.
+ */
+public abstract class SubCommandBase {
+
+  @Parameter(names = {"--raftGroup",
+      "-g"}, description = "Raft group identifier")
+  protected String raftGroupId = "demoRaftGroup123";
+
+  @Parameter(names = {"--peers", "-r"}, description =
+      "Raft peers (format: name:host:port,"
+          + "name:host:port)", required = true)
+  protected String peers;
+
+  public static RaftPeer[] parsePeers(String peers) {
+    return Stream.of(peers.split(",")).map(address -> {
+      String[] addressParts = address.split(":");
+      return new RaftPeer(RaftPeerId.valueOf(addressParts[0]),
+          addressParts[1] + ":" + addressParts[2]);
+    }).toArray(RaftPeer[]::new);
+  }
+
+  public RaftPeer[] getPeers() {
+    return parsePeers(peers);
+  }
+
+  public abstract void run() throws Exception;
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/FileStoreClient.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/FileStoreClient.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/FileStoreClient.java
index 49feeb8..c671dee 100644
--- a/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/FileStoreClient.java
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/FileStoreClient.java
@@ -60,6 +60,10 @@ public class FileStoreClient implements Closeable {
         .build();
   }
 
+  public FileStoreClient(RaftClient client) {
+    this.client = client;
+  }
+
   @Override
   public void close() throws IOException {
     client.close();

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Client.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Client.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Client.java
new file mode 100644
index 0000000..63de425
--- /dev/null
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Client.java
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.examples.filestore.cli;
+
+import org.apache.ratis.RaftConfigKeys;
+import org.apache.ratis.client.RaftClient;
+import org.apache.ratis.client.RaftClientConfigKeys;
+import org.apache.ratis.conf.Parameters;
+import org.apache.ratis.conf.RaftProperties;
+import org.apache.ratis.examples.common.SubCommandBase;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.grpc.GrpcFactory;
+import org.apache.ratis.protocol.ClientId;
+import org.apache.ratis.protocol.RaftGroup;
+import org.apache.ratis.protocol.RaftGroupId;
+import org.apache.ratis.rpc.SupportedRpcType;
+import org.apache.ratis.server.RaftServerConfigKeys;
+import org.apache.ratis.thirdparty.com.google.protobuf.ByteString;
+import org.apache.ratis.util.SizeInBytes;
+import org.apache.ratis.util.TimeDuration;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client to connect filestore example cluster.
+ */
+public abstract class Client extends SubCommandBase {
+
+
+  @Override
+  public void run() throws Exception {
+    int raftSegmentPreallocatedSize = 1024 * 1024 * 1024;
+    RaftProperties raftProperties = new RaftProperties();
+    RaftConfigKeys.Rpc.setType(raftProperties, SupportedRpcType.GRPC);
+    GrpcConfigKeys.setMessageSizeMax(raftProperties,
+        SizeInBytes.valueOf(raftSegmentPreallocatedSize));
+    RaftServerConfigKeys.Log.Appender.setBufferByteLimit(raftProperties,
+        SizeInBytes.valueOf(raftSegmentPreallocatedSize));
+    RaftServerConfigKeys.Log.setWriteBufferSize(raftProperties,
+        SizeInBytes.valueOf(raftSegmentPreallocatedSize));
+    RaftServerConfigKeys.Log.setPreallocatedSize(raftProperties,
+        SizeInBytes.valueOf(raftSegmentPreallocatedSize));
+    RaftServerConfigKeys.Log.setSegmentSizeMax(raftProperties,
+        SizeInBytes.valueOf(1 * 1024 * 1024 * 1024));
+
+    RaftServerConfigKeys.Log.setMaxCachedSegmentNum(raftProperties, 2);
+
+    RaftClientConfigKeys.Rpc.setRequestTimeout(raftProperties,
+        TimeDuration.valueOf(50000, TimeUnit.MILLISECONDS));
+    RaftClientConfigKeys.Async.setSchedulerThreads(raftProperties, 10);
+    RaftClientConfigKeys.Async.setMaxOutstandingRequests(raftProperties, 1000);
+
+
+    final RaftGroup raftGroup = RaftGroup.valueOf(RaftGroupId.valueOf(ByteString.copyFromUtf8(raftGroupId)),
+        parsePeers(peers));
+
+    RaftClient.Builder builder =
+        RaftClient.newBuilder().setProperties(raftProperties);
+    builder.setRaftGroup(raftGroup);
+    builder.setClientRpc(new GrpcFactory(new Parameters()).newRaftClientRpc(ClientId.randomId(),
raftProperties));
+    RaftClient client = builder.build();
+
+    operation(client);
+  }
+
+  protected abstract void operation(RaftClient client) throws IOException;
+}

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/FileStore.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/FileStore.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/FileStore.java
new file mode 100644
index 0000000..8cedf8b
--- /dev/null
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/FileStore.java
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ratis.examples.filestore.cli;
+
+import org.apache.ratis.examples.common.SubCommandBase;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This class enumerates all the commands enqueued by FileStore state machine.
+ */
+public class FileStore {
+  public static List<SubCommandBase> getSubCommands() {
+    List<SubCommandBase> commands = new ArrayList<>();
+    commands.add(new Server());
+    commands.add(new LoadGen());
+    return commands;
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/LoadGen.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/LoadGen.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/LoadGen.java
new file mode 100644
index 0000000..2658324
--- /dev/null
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/LoadGen.java
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.examples.filestore.cli;
+
+import com.beust.jcommander.Parameter;
+import com.beust.jcommander.Parameters;
+import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.ratis.client.RaftClient;
+import org.apache.ratis.examples.filestore.FileStoreClient;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Subcommand to generate load in filestore state machine.
+ */
+@Parameters(commandDescription = "Load Generator for FileStore")
+public class LoadGen extends Client {
+
+  private static final String UTF8_CSN = StandardCharsets.UTF_8.name();
+
+  @Parameter(names = {"--size"}, description = "Size of each file", required = true)
+  String size;
+
+  @Parameter(names = {"--numFiles"}, description = "Number of files", required = true)
+  String numFiles;
+
+  private static byte[] string2Bytes(String str) {
+    try {
+      return str.getBytes(UTF8_CSN);
+    } catch (UnsupportedEncodingException e) {
+      throw new IllegalArgumentException("UTF8 decoding is not supported", e);
+    }
+  }
+
+  @Override
+  protected void operation(RaftClient client) throws IOException {
+    int length = Integer.parseInt(size);
+    int num = Integer.parseInt(numFiles);
+    AtomicLong totalBytes = new AtomicLong(0);
+    String entropy = RandomStringUtils.randomAlphanumeric(10);
+
+    byte[] fileValue = string2Bytes(RandomStringUtils.randomAscii(length));
+    FileStoreClient fileStoreClient = new FileStoreClient(client);
+
+    System.out.println("Starting load now ");
+    long startTime = System.currentTimeMillis();
+    List<CompletableFuture<Long>> futures = new ArrayList<>();
+    for (int i = 0; i < num; i++) {
+      String path = "file-" + entropy + "-" + i;
+      ByteBuffer b = ByteBuffer.wrap(fileValue);
+      futures.add(fileStoreClient.writeAsync(path, 0, true, b));
+    }
+
+    for (CompletableFuture<Long> future : futures) {
+      Long writtenLen = future.join();
+      totalBytes.addAndGet(writtenLen);
+      if (writtenLen != length) {
+        System.out.println("File length written is wrong: " + writtenLen + length);
+      }
+    }
+    long endTime = System.currentTimeMillis();
+
+    System.out.println("Total files written: " + futures.size());
+    System.out.println("Each files size: " + length);
+    System.out.println("Total data written: " + totalBytes + " bytes");
+    System.out.println("Total time taken: " + (endTime - startTime) + " millis");
+
+    client.close();
+    System.exit(0);
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/09cc126d/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Server.java
----------------------------------------------------------------------
diff --git a/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Server.java
b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Server.java
new file mode 100644
index 0000000..0788e11
--- /dev/null
+++ b/ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Server.java
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.examples.filestore.cli;
+
+import com.beust.jcommander.Parameter;
+import com.beust.jcommander.Parameters;
+import org.apache.ratis.conf.ConfUtils;
+import org.apache.ratis.conf.RaftProperties;
+import org.apache.ratis.examples.common.SubCommandBase;
+import org.apache.ratis.examples.filestore.FileStoreCommon;
+import org.apache.ratis.examples.filestore.FileStoreStateMachine;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.protocol.RaftGroup;
+import org.apache.ratis.protocol.RaftGroupId;
+import org.apache.ratis.protocol.RaftPeer;
+import org.apache.ratis.protocol.RaftPeerId;
+import org.apache.ratis.server.RaftServer;
+import org.apache.ratis.server.RaftServerConfigKeys;
+import org.apache.ratis.statemachine.StateMachine;
+import org.apache.ratis.thirdparty.com.google.protobuf.ByteString;
+import org.apache.ratis.util.LifeCycle;
+import org.apache.ratis.util.NetUtils;
+
+import java.io.File;
+import java.util.Collections;
+import java.util.Objects;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Class to start a ratis arithmetic example server.
+ */
+@Parameters(commandDescription = "Start an filestore server")
+public class Server extends SubCommandBase {
+
+  @Parameter(names = {"--id", "-i"}, description = "Raft id of this server", required = true)
+  private String id;
+
+  @Parameter(names = {"--storage", "-s"}, description = "Storage dir", required = true)
+  private File storageDir;
+
+
+  @Override
+  public void run() throws Exception {
+    RaftPeerId peerId = RaftPeerId.valueOf(id);
+    RaftProperties properties = new RaftProperties();
+
+    RaftPeer[] peers = getPeers();
+    final int port = NetUtils.createSocketAddr(getPeer(peerId).getAddress()).getPort();
+    GrpcConfigKeys.Server.setPort(properties, port);
+    properties.setInt(GrpcConfigKeys.OutputStream.RETRY_TIMES_KEY, Integer.MAX_VALUE);
+    RaftServerConfigKeys.setStorageDirs(properties, Collections.singletonList(storageDir));
+    ConfUtils.setFile(properties::setFile, FileStoreCommon.STATEMACHINE_DIR_KEY,
+        storageDir);
+    StateMachine stateMachine = new FileStoreStateMachine(properties);
+
+    final RaftGroup raftGroup = RaftGroup.valueOf(RaftGroupId.valueOf(ByteString.copyFromUtf8(raftGroupId)),
peers);
+    RaftServer raftServer = RaftServer.newBuilder()
+        .setServerId(RaftPeerId.valueOf(id))
+        .setStateMachine(stateMachine).setProperties(properties)
+        .setGroup(raftGroup)
+        .build();
+    raftServer.start();
+
+    for(; raftServer.getLifeCycleState() != LifeCycle.State.CLOSED;) {
+      TimeUnit.SECONDS.sleep(1);
+    }
+  }
+
+  /**
+   * @return the peer with the given id if it is in this group; otherwise, return null.
+   */
+  public RaftPeer getPeer(RaftPeerId id) {
+    Objects.requireNonNull(id, "id == null");
+    for (RaftPeer p : getPeers()) {
+      if (id.equals(p.getId())) {
+        return p;
+      }
+    }
+    throw new IllegalArgumentException("Raft peer id " + id + " is not part of the raft group
definitions " + peers);
+  }
+}



Mime
View raw message