tinkerpop-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From twil...@apache.org
Subject incubator-tinkerpop git commit: This commit includes the following: * added new gremlin-benchmark module and initial benchmark classes * deprecated the JUnit benchmark classes, noted this deprecation in upgrade doc * expanded scope of LoadGraphWith annot
Date Tue, 29 Mar 2016 04:03:18 GMT
Repository: incubator-tinkerpop
Updated Branches:
  refs/heads/master d5a785603 -> 682d313e8


This commit includes the following:
* added new gremlin-benchmark module and initial benchmark classes
* deprecated the JUnit benchmark classes, noted this deprecation in upgrade doc
* expanded scope of LoadGraphWith annotation to allow for class level annotation
* added developer benchmarking documentation


Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/682d313e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/682d313e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/682d313e

Branch: refs/heads/master
Commit: 682d313e8632c10470f0376f38c169acb6e047c3
Parents: d5a7856
Author: Ted Wilmes <twilmes@gmail.com>
Authored: Mon Mar 28 21:12:57 2016 -0500
Committer: Ted Wilmes <twilmes@gmail.com>
Committed: Mon Mar 28 21:12:57 2016 -0500

----------------------------------------------------------------------
 .gitignore                                      |   1 +
 CHANGELOG.asciidoc                              |   1 +
 docs/src/dev/developer/contributing.asciidoc    |  31 +++++
 .../upgrade/release-3.2.x-incubating.asciidoc   |   5 +
 gremlin-benchmark/pom.xml                       | 130 +++++++++++++++++++
 .../benchmark/util/AbstractBenchmarkBase.java   | 117 +++++++++++++++++
 .../benchmark/util/AbstractGraphBenchmark.java  |  85 ++++++++++++
 .../util/AbstractGraphMutateBenchmark.java      |  48 +++++++
 .../gremlin/process/GraphMutateBenchmark.java   |  92 +++++++++++++
 .../process/GraphTraversalBenchmark.java        | 101 ++++++++++++++
 .../apache/tinkerpop/gremlin/LoadGraphWith.java |   2 +-
 .../process/ProcessPerformanceSuite.java        |   3 +
 .../process/TraversalPerformanceTest.java       |   3 +
 pom.xml                                         |   8 +-
 14 files changed, 625 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index f4779c5..11ca63e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -14,3 +14,4 @@ _bsp/
 doc/out
 docs/*.asciidoc
 ext/
+benchmarks/

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index ea0d52b..87a7ba6 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -26,6 +26,7 @@ image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/
 TinkerPop 3.2.0 (NOT OFFICIALLY RELEASED YET)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+* Added `gremlin-benchmark` module with JMH benchmarking base classes that can be used for
further benchmark development.
 * `TraversalStrategies.GlobalCache` supports both `Graph` and `GraphComputer` strategy registrations.
 * `select("a","b").by("name").by("age")`-style traversals now work in OLAP with new `PathProcessorStrategy`.
 * `DedupGlobalStep` can now handle star-bound `by()`-modulators and scoped keys on `GraphComputer`.

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/docs/src/dev/developer/contributing.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/dev/developer/contributing.asciidoc b/docs/src/dev/developer/contributing.asciidoc
index 4a76184..86a90ba 100644
--- a/docs/src/dev/developer/contributing.asciidoc
+++ b/docs/src/dev/developer/contributing.asciidoc
@@ -89,6 +89,7 @@ mvn -Dmaven.javadoc.skip=true --projects tinkergraph-gremlin test
 ** Execute with the `-DincludeNeo4j` option to include transactional tests.
 ** Execute with the `-DuseEpoll` option to try to use Netty native transport (works on Linux,
but will fallback to Java NIO on other OS).
 * Performance Tests: `mvn verify -DskipPerformanceTests=false`
+* Benchmarks: `mvn verify -DskipBenchmarks=false`
 
 IDE Setup with Intellij
 -----------------------
@@ -301,6 +302,36 @@ When writing a test case for a Gremlin step, be sure to use the following
conven
 ** `checkResults(Arrays.asList("marko","josh"), traversal)`
 ** `checkMap(new HashMap<String,Long>() {{ put("marko",1l); }}, traversal.next())`
 
+Developing Benchmarks
+~~~~~~~~~~~~~~~~~~~~~
+
+Benchmarks are a useful tool to track performance between TinkerPop versions and also as
tools to aid development
+decision making. TinkerPop uses link:http://openjdk.java.net/projects/code-tools/jmh/[OpenJDK
JMH] for benchmark development.
+The JMH framework provides tools for writing robust benchmarking code that avoid many of
the pitfalls inherent in benchmarking
+JIT compiled code on the JVM.  Example JMH benchmarks can be found
+link:http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/[here].
+
+TinkerPop benchmarks live in the `gremlin-benchmark` module and can either be run from within
your IDE or as a standalone
+uber-jar.  The uber-jar is the JMH recommended approach and also makes it easy to distribute
artifacts to various environments
+to gather benchmarking numbers.  Having said that, in most cases it should be sufficient
to run it from within the IDE.
+
+Benchmarks will not run by default because they are time consuming.  To enable benchmarks
during the test phase do
+`-DskipBenchmarks=false`.  To change the number of warmup iterations, measurement iterations,
and forks you can do
+`mvn clean test -DskipBenchmarks=false -DdefaultForks=5 -DmeasureIterations=20 -DwarmupIterations=20`.
 Benchmark results
+will be output by default to the `benchmarks` directory in JSON format.
+
+Benchmarks may also be run from the command line using the JMH runner.  Build the uber-jar
and simply run
+`java -jar gremlin-benchmark-TP-VERSION.jar`.  To see a list of JMH runner options, add the
`-h` flag.
+
+The JUnit/JMH integration was inspired by the Netty projects microbenchmarking suite.  Please
refer to the Netty
+link:http://netty.io/wiki/microbenchmarks.html[docs] for more details.  Presently there are
3 abstract benchmark classes
+that may be used as building blocks for your benchmarks; `AbstractBenchmarkBase`, `AbstractGraphBenchmark`,
and
+`AbstractGraphMutateBenchmark`.
+
+* `AbstractBenchmarkBase` - extend when your benchmark does not require a graph instance
+* `AbstractGraphBenchmark` - extend when you are benchmarking read operations against a graph
+* `AbstractGraphMutateBenchmark` - extend when you are benchmarking graph mutation operations
eg. `g.addV()`, `graph.addVertex()`
+
 [[rtc]]
 Review then Commit
 ~~~~~~~~~~~~~~~~~~

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/docs/src/upgrade/release-3.2.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade/release-3.2.x-incubating.asciidoc b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
index d28669e..8d53287 100644
--- a/docs/src/upgrade/release-3.2.x-incubating.asciidoc
+++ b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
@@ -322,6 +322,11 @@ no longer exists and such post-reduction processing is handled by the
reducing s
 
 See: link:https://issues.apache.org/jira/browse/TINKERPOP-1164[TINKERPOP-1164]
 
+Performance Tests
++++++++++++++++++
+The `ProcessPerformanceSuite` and `TraversalPerformanceTest` have been deprecated.  They
are still available, but going forward,
+providers should implement their own performance tests and not rely on the built-in JUnit
benchmark-based performance test suit.
+
 Graph Processor Providers
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/gremlin-benchmark/pom.xml
----------------------------------------------------------------------
diff --git a/gremlin-benchmark/pom.xml b/gremlin-benchmark/pom.xml
new file mode 100644
index 0000000..092e6d5
--- /dev/null
+++ b/gremlin-benchmark/pom.xml
@@ -0,0 +1,130 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <artifactId>tinkerpop</artifactId>
+        <groupId>org.apache.tinkerpop</groupId>
+        <version>3.2.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>gremlin-benchmark</artifactId>
+    <name>Apache TinkerPop :: Gremlin Benchmark</name>
+    <properties>
+        <jmh.version>1.11.3</jmh.version>
+        <!-- Skip benchmarks by default because they are time consuming. -->
+        <skipBenchmarks>true</skipBenchmarks>
+        <skipTests>${skipBenchmarks}</skipTests>
+    </properties>
+
+    <dependencies>
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <version>${junit.version}</version>
+            <scope>compile</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.tinkerpop</groupId>
+            <artifactId>gremlin-core</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.tinkerpop</groupId>
+            <artifactId>gremlin-test</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.tinkerpop</groupId>
+            <artifactId>tinkergraph-gremlin</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.openjdk.jmh</groupId>
+            <artifactId>jmh-core</artifactId>
+            <version>${jmh.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.openjdk.jmh</groupId>
+            <artifactId>jmh-generator-annprocess</artifactId>
+            <version>${jmh.version}</version>
+            <scope>provided</scope>
+        </dependency>
+    </dependencies>
+    <build>
+        <plugins>
+            <plugin>
+                <artifactId>maven-deploy-plugin</artifactId>
+                <configuration>
+                    <skip>true</skip>
+                </configuration>
+            </plugin>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-surefire-plugin</artifactId>
+                <configuration>
+                    <testSourceDirectory>${project.build.sourceDirectory}</testSourceDirectory>
+                    <testClassesDirectory>${project.build.outputDirectory}</testClassesDirectory>
+                    <includes>
+                        <include>**/*Benchmark*.java</include>
+                    </includes>
+                    <excludes>
+                        <exclude>**/*$*.class</exclude>
+                        <exclude>**/Abstract*</exclude>
+                        <exclude>**/*_jmhType*</exclude>
+                    </excludes>
+                    <systemPropertyVariables>
+                        <benchmarkReportDir>${project.build.directory}/reports/benchmark/</benchmarkReportDir>
+                    </systemPropertyVariables>
+                </configuration>
+            </plugin>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-shade-plugin</artifactId>
+                <version>2.2</version>
+                <executions>
+                    <execution>
+                        <phase>package</phase>
+                        <goals>
+                            <goal>shade</goal>
+                        </goals>
+                        <configuration>
+                            <createDependencyReducedPom>false</createDependencyReducedPom>
+                            <transformers>
+                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+                                    <mainClass>org.openjdk.jmh.Main</mainClass>
+                                </transformer>
+                            </transformers>
+                            <filters>
+                                <filter>
+                                    <artifact>*:*</artifact>
+                                    <excludes>
+                                        <exclude>META-INF/*.SF</exclude>
+                                        <exclude>META-INF/*.DSA</exclude>
+                                        <exclude>META-INF/*.RSA</exclude>
+                                    </excludes>
+                                </filter>
+                            </filters>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+</project>

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractBenchmarkBase.java
----------------------------------------------------------------------
diff --git a/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractBenchmarkBase.java
b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractBenchmarkBase.java
new file mode 100644
index 0000000..5b73045
--- /dev/null
+++ b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractBenchmarkBase.java
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.benchmark.util;
+
+import org.junit.Test;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.results.format.ResultFormatType;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.ChainedOptionsBuilder;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+import java.io.File;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+/**
+ * Base class for all TinkerPop OpenJDK JMH benchmarks.  Based upon Netty's approach to running
JMH benchmarks
+ * from JUnit.
+ *
+ * @see <a href="http://netty.io/wiki/microbenchmarks.html"</a>
+ *
+ * @author Ted Wilmes (http://twilmes.org)
+ */
+@Warmup(iterations = AbstractBenchmarkBase.DEFAULT_WARMUP_ITERATIONS)
+@Measurement(iterations = AbstractBenchmarkBase.DEFAULT_MEASURE_ITERATIONS)
+@Fork(AbstractBenchmarkBase.DEFAULT_FORKS)
+public abstract class AbstractBenchmarkBase {
+
+    protected static final int DEFAULT_WARMUP_ITERATIONS = 10;
+    protected static final int DEFAULT_MEASURE_ITERATIONS = 10;
+    protected static final int DEFAULT_FORKS = 2;
+    protected static final String DEFAULT_BENCHMARK_DIRECTORY = "./benchmarks/";
+
+    protected static final String[] JVM_ARGS = {
+            "-server", "-Xms1g", "-Xmx1g"
+    };
+
+    @Test
+    public void run() throws Exception {
+        final String className = getClass().getSimpleName();
+
+        final ChainedOptionsBuilder runnerOptions = new OptionsBuilder()
+                .include(".*" + className + ".*")
+                .jvmArgs(JVM_ARGS);
+
+        if (getWarmupIterations() > 0) {
+            runnerOptions.warmupIterations(getWarmupIterations());
+        }
+
+        if (getMeasureIterations() > 0) {
+            runnerOptions.measurementIterations(getMeasureIterations());
+        }
+
+        if (getForks() > 0) {
+            runnerOptions.forks(getForks());
+        }
+
+        if (getReportDir() != null) {
+            final String dtmStr = new SimpleDateFormat("yyyyMMddHHmmss").format(new Date());
+            final String filePath = getReportDir() + className + "-" + dtmStr + ".json";
+            final File file = new File(filePath);
+            if (file.exists()) {
+                file.delete();
+            } else {
+                file.getParentFile().mkdirs();
+                file.createNewFile();
+            }
+
+            runnerOptions.resultFormat(ResultFormatType.JSON);
+            runnerOptions.result(filePath);
+        }
+
+        new Runner(runnerOptions.build()).run();
+    }
+
+    protected int getWarmupIterations() {
+        return getIntProperty("warmupIterations", DEFAULT_WARMUP_ITERATIONS);
+    }
+
+    protected int getMeasureIterations() {
+        return getIntProperty("measureIterations", DEFAULT_MEASURE_ITERATIONS);
+    }
+
+    protected int getForks() {
+        return getIntProperty("forks", DEFAULT_FORKS);
+    }
+
+    protected String getReportDir() {
+        return System.getProperty("benchmark.dir", DEFAULT_BENCHMARK_DIRECTORY);
+    }
+
+    private int getIntProperty(final String propertyName, final int defaultValue) {
+        final String propertyValue = System.getProperty(propertyName);
+        if(propertyValue == null) {
+            return defaultValue;
+        }
+        return Integer.valueOf(propertyValue);
+    }
+}

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractGraphBenchmark.java
----------------------------------------------------------------------
diff --git a/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractGraphBenchmark.java
b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractGraphBenchmark.java
new file mode 100644
index 0000000..ba0105c
--- /dev/null
+++ b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractGraphBenchmark.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.benchmark.util;
+
+import org.apache.tinkerpop.gremlin.LoadGraphWith;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.io.GraphReader;
+import org.apache.tinkerpop.gremlin.structure.io.gryo.GryoReader;
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
+
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * Read-only graph benchmarks extend {@code AbstractGraphBenchmark}.  Annotating your benchmark
class with {@link LoadGraphWith}
+ * will load the {@link TinkerGraph} instance with the desired data set.
+ *
+ * @author Ted Wilmes (http://twilmes.org)
+ */
+@State(Scope.Thread)
+public class AbstractGraphBenchmark extends AbstractBenchmarkBase {
+
+    private final String PATH = "/org/apache/tinkerpop/gremlin/structure/io/gryo/";
+
+    protected Graph graph;
+    protected GraphTraversalSource g;
+
+    /**
+     * Opens a new {@link TinkerGraph} instance and optionally preloads it with one of the
test data sets enumerated
+     * in {@link LoadGraphWith}.
+     *
+     * @throws IOException on failure to load graph
+     */
+    @Setup
+    public void prepare() throws IOException {
+        graph = TinkerGraph.open();
+        g = graph.traversal();
+
+        final LoadGraphWith[] loadGraphWiths = this.getClass().getAnnotationsByType(LoadGraphWith.class);
+        final LoadGraphWith loadGraphWith = loadGraphWiths.length == 0 ? null : loadGraphWiths[0];
+        final LoadGraphWith.GraphData loadGraphWithData = null == loadGraphWith ? null :
loadGraphWith.value();
+
+        String graphFile;
+        if(loadGraphWithData != null) {
+            if (loadGraphWithData.equals(LoadGraphWith.GraphData.GRATEFUL)) {
+                graphFile = "grateful-dead.kryo";
+            } else if (loadGraphWithData.equals(LoadGraphWith.GraphData.MODERN)) {
+                graphFile = "tinkerpop-modern.kryo";
+            } else if (loadGraphWithData.equals(LoadGraphWith.GraphData.CLASSIC)) {
+                graphFile = "tinkerpop-classic.kryo";
+            } else if (loadGraphWithData.equals(LoadGraphWith.GraphData.CREW)) {
+                graphFile = "tinkerpop-crew.kryo";
+            } else {
+                throw new RuntimeException("Could not load graph with " + loadGraphWithData);
+            }
+
+            final GraphReader reader = GryoReader.build().create();
+            try (final InputStream stream = AbstractGraphBenchmark.class.
+                    getResourceAsStream(PATH + graphFile)) {
+                reader.readGraph(stream, graph);
+            }
+        }
+    }
+}

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractGraphMutateBenchmark.java
----------------------------------------------------------------------
diff --git a/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractGraphMutateBenchmark.java
b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractGraphMutateBenchmark.java
new file mode 100644
index 0000000..79d9916
--- /dev/null
+++ b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/benchmark/util/AbstractGraphMutateBenchmark.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.benchmark.util;
+
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
+
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+
+/**
+ * Graph write and update benchmarks extend {@code AbstractGraphMutateBenchmark}.  {@code
AbstractGraphMutateBenchmark}
+ * runs setup once per invocation so that benchmark measurements are made on an empty {@link
TinkerGraph}.  This approach
+ * was taken to isolate the tested method from the performance side effects of unbounded
graph growth.
+ *
+ * @author Ted Wilmes (http://twilmes.org)
+ */
+@State(Scope.Thread)
+public abstract class AbstractGraphMutateBenchmark extends AbstractBenchmarkBase {
+
+    protected Graph graph;
+    protected GraphTraversalSource g;
+
+    @Setup(Level.Invocation)
+    public void prepare() {
+        graph = TinkerGraph.open();
+        g = graph.traversal();
+    }
+}

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/gremlin-benchmark/src/main/java/org/apache/tinkerpop/gremlin/process/GraphMutateBenchmark.java
----------------------------------------------------------------------
diff --git a/gremlin-benchmark/src/main/java/org/apache/tinkerpop/gremlin/process/GraphMutateBenchmark.java
b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/gremlin/process/GraphMutateBenchmark.java
new file mode 100644
index 0000000..04d7138
--- /dev/null
+++ b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/gremlin/process/GraphMutateBenchmark.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.process;
+
+import org.apache.tinkerpop.benchmark.util.AbstractGraphMutateBenchmark;
+import org.apache.tinkerpop.gremlin.structure.Edge;
+import org.apache.tinkerpop.gremlin.structure.Property;
+import org.apache.tinkerpop.gremlin.structure.Vertex;
+import org.apache.tinkerpop.gremlin.structure.VertexProperty;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Setup;
+
+/**
+ * {@code GraphMutateBenchmark} benchmarks {@link org.apache.tinkerpop.gremlin.process.traversal.Traversal}
and
+ * {@link org.apache.tinkerpop.gremlin.structure.Graph} mutation methods.
+ *
+ * @author Ted Wilmes (http://twilmes.org)
+ */
+public class GraphMutateBenchmark extends AbstractGraphMutateBenchmark {
+
+    private Vertex a;
+    private Vertex b;
+    private Vertex c;
+    private Edge e;
+
+    @Setup
+    @Override
+    public void prepare() {
+        super.prepare();
+        a = g.addV().next();
+        b = g.addV().next();
+        c = g.addV().next();
+        e = b.addEdge("knows", c);
+    }
+
+    @Benchmark
+    public Vertex testAddVertex() {
+        return graph.addVertex("test");
+    }
+
+    @Benchmark
+    public VertexProperty testVertexProperty() {
+        return a.property("name", "Susan");
+    }
+
+    @Benchmark
+    public Edge testAddEdge() {
+        return a.addEdge("knows", b);
+    }
+
+    @Benchmark
+    public Property testEdgeProperty() {
+        return e.property("met", 1967);
+    }
+
+    @Benchmark
+    public Vertex testAddV() {
+        return g.addV("test").next();
+    }
+
+    @Benchmark
+    public Vertex testVertexPropertyStep() {
+        return g.V(a).property("name", "Susan").next();
+    }
+
+    @Benchmark
+    public Edge testAddE() {
+        return g.V(a).as("a").V(b).as("b").addE("knows").from("a").to("b").next();
+    }
+
+    @Benchmark
+    public Edge testEdgePropertyStep() {
+        return g.E(e).property("met", 1967).next();
+    }
+}

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/gremlin-benchmark/src/main/java/org/apache/tinkerpop/gremlin/process/GraphTraversalBenchmark.java
----------------------------------------------------------------------
diff --git a/gremlin-benchmark/src/main/java/org/apache/tinkerpop/gremlin/process/GraphTraversalBenchmark.java
b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/gremlin/process/GraphTraversalBenchmark.java
new file mode 100644
index 0000000..191f5e7
--- /dev/null
+++ b/gremlin-benchmark/src/main/java/org/apache/tinkerpop/gremlin/process/GraphTraversalBenchmark.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.process;
+
+import org.apache.tinkerpop.benchmark.util.AbstractGraphBenchmark;
+import org.apache.tinkerpop.gremlin.LoadGraphWith;
+import org.apache.tinkerpop.gremlin.process.traversal.Path;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
+import org.apache.tinkerpop.gremlin.structure.Edge;
+import org.apache.tinkerpop.gremlin.structure.Vertex;
+
+import org.openjdk.jmh.annotations.Benchmark;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.out;
+
+/**
+ * Runs a traversal benchmarks against a {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph}
loaded
+ * with the Grateful Dead data set.
+ *
+ * @author Ted Wilmes (http://twilmes.org)
+ */
+@LoadGraphWith(LoadGraphWith.GraphData.GRATEFUL)
+public class GraphTraversalBenchmark extends AbstractGraphBenchmark {
+
+    @Benchmark
+    public List<Vertex> g_V_outE_inV_outE_inV_outE_inV() throws Exception {
+        return g.V().outE().inV().outE().inV().outE().inV().toList();
+    }
+
+    @Benchmark
+    public List<Vertex> g_V_out_out_out() throws Exception {
+        return g.V().out().out().out().toList();
+    }
+
+    @Benchmark
+    public List<Path> g_V_out_out_out_path() throws Exception {
+        return g.V().out().out().out().path().toList();
+    }
+
+    @Benchmark
+    public List<Vertex> g_V_repeatXoutX_timesX2X() throws Exception {
+        return g.V().repeat(out()).times(2).toList();
+    }
+
+    @Benchmark
+    public List<Vertex> g_V_repeatXoutX_timesX3X() throws Exception {
+        return g.V().repeat(out()).times(3).toList();
+    }
+
+    @Benchmark
+    public List<List<Object>> g_V_localXout_out_valuesXnameX_foldX() throws Exception
{
+        return g.V().local(out().out().values("name").fold()).toList();
+    }
+
+    @Benchmark
+    public List<List<Object>> g_V_out_localXout_out_valuesXnameX_foldX() throws
Exception {
+        return g.V().out().local(out().out().values("name").fold()).toList();
+    }
+
+    @Benchmark
+    public List<List<Object>> g_V_out_mapXout_out_valuesXnameX_toListX() throws
Exception {
+        return g.V().out().map(v -> g.V(v.get()).out().out().values("name").toList()).toList();
+    }
+
+    @Benchmark
+    public List<Map<Object, Long>> g_V_label_groupCount() throws Exception {
+        return g.V().label().groupCount().toList();
+    }
+
+    @Benchmark
+    public List<Object> g_V_match_selectXbX_valuesXnameX() throws Exception {
+        return g.V().match(
+                __.as("a").has("name", "Garcia"),
+                __.as("a").in("writtenBy").as("b"),
+                __.as("a").in("sungBy").as("b")).select("b").values("name").toList();
+    }
+
+    @Benchmark
+    public List<Edge> g_E_hasLabelXwrittenByX_whereXinV_inEXsungByX_count_isX0XX_subgraphXsgX()
throws Exception {
+        return g.E().hasLabel("writtenBy").where(__.inV().inE("sungBy").count().is(0)).subgraph("sg").toList();
+    }
+}

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/LoadGraphWith.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/LoadGraphWith.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/LoadGraphWith.java
index 17893e5..071d579 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/LoadGraphWith.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/LoadGraphWith.java
@@ -52,7 +52,7 @@ import static org.apache.tinkerpop.gremlin.structure.Graph.Features.VertexFeatur
  */
 @Inherited
 @Retention(RetentionPolicy.RUNTIME)
-@Target(ElementType.METHOD)
+@Target({ElementType.TYPE, ElementType.METHOD})
 public @interface LoadGraphWith {
 
     public enum GraphData {

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessPerformanceSuite.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessPerformanceSuite.java
b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessPerformanceSuite.java
index f787450..1b2c36c 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessPerformanceSuite.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/ProcessPerformanceSuite.java
@@ -36,7 +36,10 @@ import org.junit.runners.model.RunnerBuilder;
  * For more information on the usage of this suite, please see {@link StructureStandardSuite}.
  *
  * @author Stephen Mallette (http://stephen.genoprime.com)
+ *
+ * @deprecated  As of release 3.2.0.  Provider performance tests may be implemented as needed
by providers and will not be included as part of the TinkerPop distribution.
  */
+@Deprecated
 public class ProcessPerformanceSuite extends AbstractGremlinSuite {
 
     /**

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/TraversalPerformanceTest.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/TraversalPerformanceTest.java
b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/TraversalPerformanceTest.java
index b78d4bf..be750f4 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/TraversalPerformanceTest.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/TraversalPerformanceTest.java
@@ -36,10 +36,13 @@ import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.out;
 /**
  * @author Stephen Mallette (http://stephen.genoprime.com)
  * @author Daniel Kuppitz (http://gremlin.guru)
+ *
+ * @deprecated  As of release 3.2.0.  Provider performance tests may be implemented as needed
by providers and will not be included as part of the TinkerPop distribution.
  */
 @AxisRange(min = 0, max = 1)
 @BenchmarkMethodChart(filePrefix = "gremlin-traversal")
 @BenchmarkHistoryChart(labelWith = LabelType.CUSTOM_KEY, maxRuns = 20, filePrefix = "hx-gremlin-traversal")
+@Deprecated
 public class TraversalPerformanceTest extends AbstractGremlinTest {
 
     public final static int DEFAULT_BENCHMARK_ROUNDS = 10;

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/682d313e/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 98e7553..4937bdd 100644
--- a/pom.xml
+++ b/pom.xml
@@ -103,11 +103,17 @@ limitations under the License.
             <email>dylan.millikin@gmail.com</email>
             <url>https://github.com/PommeVerte</url>
         </contributor>
+        <contributor>
+            <name>Ted Wilmes</name>
+            <email>twilmes@gmail.com</email>
+            <url>https://github.com/twilmes</url>
+        </contributor>
     </contributors>
     <modules>
         <module>gremlin-shaded</module>
         <module>gremlin-core</module>
         <module>gremlin-test</module>
+        <module>gremlin-benchmark</module>
         <module>gremlin-groovy</module>
         <module>gremlin-groovy-test</module>
         <module>tinkergraph-gremlin</module>
@@ -874,7 +880,7 @@ limitations under the License.
                                     <overview>${basedir}/docs/javadoc/overview.html</overview>
                                     <quiet>true</quiet>
                                     <sourcepath>
-                                        giraph-gremlin/src/main/java:gremlin-core/src/main/java:gremlin-driver/src/main/java:gremlin-groovy/src/main/java:gremlin-groovy-test/src/main/java:gremlin-server/src/main/java:gremlin-test/src/main/java:hadoop-gremlin/src/main/java:neo4j-gremlin/src/main/java:spark-gremlin/src/main/java:tinkergraph-gremlin/src/main/java
+                                        giraph-gremlin/src/main/java:gremlin-core/src/main/java:gremlin-driver/src/main/java:gremlin-groovy/src/main/java:gremlin-groovy-test/src/main/java:gremlin-server/src/main/java:gremlin-test/src/main/java:hadoop-gremlin/src/main/java:neo4j-gremlin/src/main/java:spark-gremlin/src/main/java:tinkergraph-gremlin/src/main/java:gremlin-benchmark/src/main/java
                                     </sourcepath>
                                 </configuration>
                             </execution>



Mime
View raw message