drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5126) Provide simplified, unified "cluster fixture" for tests
Date Wed, 18 Jan 2017 22:13:26 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828858#comment-15828858
] 

ASF GitHub Bot commented on DRILL-5126:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/710#discussion_r96739079
  
    --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/QueryBuilder.java ---
    @@ -0,0 +1,314 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + ******************************************************************************/
    +package org.apache.drill.test;
    +
    +import java.util.List;
    +
    +import org.apache.drill.PlanTestBase;
    +import org.apache.drill.QueryTestUtil;
    +import org.apache.drill.common.config.DrillConfig;
    +import org.apache.drill.common.expression.SchemaPath;
    +import org.apache.drill.exec.client.PrintingResultsListener;
    +import org.apache.drill.exec.client.QuerySubmitter.Format;
    +import org.apache.drill.exec.proto.UserBitShared.QueryId;
    +import org.apache.drill.exec.proto.UserBitShared.QueryType;
    +import org.apache.drill.exec.proto.helper.QueryIdHelper;
    +import org.apache.drill.exec.record.RecordBatchLoader;
    +import org.apache.drill.exec.record.VectorWrapper;
    +import org.apache.drill.exec.rpc.RpcException;
    +import org.apache.drill.exec.rpc.user.AwaitableUserResultsListener;
    +import org.apache.drill.exec.rpc.user.QueryDataBatch;
    +import org.apache.drill.exec.rpc.user.UserResultsListener;
    +import org.apache.drill.exec.util.VectorUtil;
    +import org.apache.drill.exec.vector.NullableVarCharVector;
    +import org.apache.drill.exec.vector.ValueVector;
    +import org.apache.drill.test.BufferingQueryEventListener.QueryEvent;
    +
    +import com.google.common.base.Preconditions;
    +
    +/**
    + * Builder for a Drill query. Provides all types of query formats,
    + * and a variety of ways to run the query.
    + */
    +
    +public class QueryBuilder {
    +
    +  /**
    +   * Summary results of a query: records, batches, run time.
    +   */
    +
    +  public static class QuerySummary {
    +    private final QueryId queryId;
    +    private final int records;
    +    private final int batches;
    +    private final long ms;
    +
    +    public QuerySummary(QueryId queryId, int recordCount, int batchCount, long elapsed)
{
    +      this.queryId = queryId;
    +      records = recordCount;
    +      batches = batchCount;
    +      ms = elapsed;
    +    }
    +
    +    public long recordCount( ) { return records; }
    +    public int batchCount( ) { return batches; }
    +    public long runTimeMs( ) { return ms; }
    +    public QueryId queryId( ) { return queryId; }
    +    public String queryIdString( ) { return QueryIdHelper.getQueryId(queryId); }
    +
    +  }
    +
    +  private final ClientFixture client;
    +  private QueryType queryType;
    +  private String queryText;
    +
    +  QueryBuilder(ClientFixture client) {
    +    this.client = client;
    +  }
    +
    +  public QueryBuilder query(QueryType type, String text) {
    +    queryType = type;
    +    queryText = text;
    +    return this;
    +  }
    +
    +  public QueryBuilder sql(String sql) {
    +    return query( QueryType.SQL, sql );
    +  }
    +
    +  public QueryBuilder sql(String query, Object... args) {
    +    return sql(String.format(query, args));
    +  }
    +
    +  public QueryBuilder physical(String plan) {
    +    return query( QueryType.PHYSICAL, plan);
    +  }
    +
    +  public QueryBuilder sqlResource(String resource) {
    +    sql(ClusterFixture.loadResource(resource));
    +    return this;
    +  }
    +
    +  public QueryBuilder sqlResource(String resource, Object... args) {
    +    sql(ClusterFixture.loadResource(resource), args);
    +    return this;
    +  }
    +
    +  public QueryBuilder physicalResource(String resource) {
    +    physical(ClusterFixture.loadResource(resource));
    +    return this;
    +  }
    +
    +  /**
    +   * Run the query returning just a summary of the results: record count,
    +   * batch count and run time. Handy when doing performance tests when the
    +   * validity of the results is verified in some other test.
    +   *
    +   * @return the query summary
    +   */
    +
    +  public QuerySummary run() {
    +    return produceSummary(withEventListener());
    +  }
    +
    +  /**
    +   * Run the query and return a list of the result batches. Use
    +   * if the batch count is small and you want to work with them.
    +   * @return a list of batches resulting from the query
    +   * @throws RpcException
    +   */
    +
    +  public List<QueryDataBatch> results() throws RpcException {
    +    Preconditions.checkNotNull(queryType, "Query not provided.");
    +    Preconditions.checkNotNull(queryText, "Query not provided.");
    +    return client.client().runQuery(queryType, queryText);
    +  }
    +
    +  /**
    +   * Run the query with the listener provided. Use when the result
    +   * count will be large, or you don't need the results.
    +   *
    +   * @param listener the Drill listener
    +   */
    +
    +  public void withListener(UserResultsListener listener) {
    +    Preconditions.checkNotNull(queryType, "Query not provided.");
    +    Preconditions.checkNotNull(queryText, "Query not provided.");
    +    client.client().runQuery(queryType, queryText, listener);
    +  }
    +
    +  /**
    +   * Run the query, return an easy-to-use event listener to process
    +   * the query results. Use when the result set is large. The listener
    +   * allows the caller to iterate over results in the test thread.
    +   * (The listener implements a producer-consumer model to hide the
    +   * details of Drill listeners.)
    +   *
    +   * @return the query event listener
    +   */
    +
    +  public BufferingQueryEventListener withEventListener( ) {
    +    BufferingQueryEventListener listener = new BufferingQueryEventListener( );
    +    withListener(listener);
    +    return listener;
    +  }
    +
    +  public long printCsv() {
    +    return print(Format.CSV);
    +  }
    +
    +  public long print( Format format ) {
    +    return print(format,20);
    +  }
    +
    +  public long print(Format format, int colWidth) {
    +    return runAndWait( new PrintingResultsListener( client.cluster().config( ), format,
colWidth ) );
    +  }
    +
    +  /**
    +   * Run a query and optionally print the output in TSV format.
    +   * Similar to {@link QueryTestUtil#test} with one query. Output is printed
    +   * only if the tests are running as verbose.
    +   *
    +   * @return the number of rows returned
    +   */
    +  public long print() {
    +    DrillConfig config = client.cluster().config( );
    +
    +    // Note: verbose check disabled until that change is
    +    // committed.
    +
    +    boolean verbose = ! config.getBoolean(QueryTestUtil.TEST_QUERY_PRINTING_SILENT) /*
||
    +                      DrillTest.verbose() */;
    +    if (verbose) {
    +      return print(Format.TSV, VectorUtil.DEFAULT_COLUMN_WIDTH);
    +    } else {
    +      return run().recordCount();
    +    }
    +  }
    +
    +  public long runAndWait(UserResultsListener listener) {
    +    AwaitableUserResultsListener resultListener =
    +        new AwaitableUserResultsListener(listener);
    +    withListener( resultListener );
    +    try {
    +      return resultListener.await();
    +    } catch (Exception e) {
    +      throw new IllegalStateException(e);
    +    }
    +  }
    +
    +  /**
    +   * Submit an "EXPLAIN" statement, and return text form of the
    +   * plan.
    +   * @throws Exception if the query fails
    +   */
    +
    +  public String explainText() throws Exception {
    +    return explain(ClusterFixture.EXPLAIN_PLAN_TEXT);
    +  }
    +
    +  /**
    +   * Submit an "EXPLAIN" statement, and return the JSON form of the
    +   * plan.
    +   * @throws Exception if the query fails
    +   */
    +
    +  public String explainJson() throws Exception {
    +    return explain(ClusterFixture.EXPLAIN_PLAN_JSON);
    +  }
    +
    +  public String explain(String format) throws Exception {
    +    queryText = "EXPLAIN PLAN FOR " + queryText;
    +    return queryPlan(format);
    +  }
    +
    +  private QuerySummary produceSummary(BufferingQueryEventListener listener) {
    +    long start = System.currentTimeMillis();
    +    int recordCount = 0;
    +    int batchCount = 0;
    +    QueryId queryId = null;
    +    loop:
    +    for ( ; ; ) {
    +      QueryEvent event = listener.get();
    --- End diff --
    
    Next commit will handle this by wrapping the `InterruptedException` in an event and returning
it. That particular exception should never occur in a test case, but no harm in being complete.


> Provide simplified, unified "cluster fixture" for tests
> -------------------------------------------------------
>
>                 Key: DRILL-5126
>                 URL: https://issues.apache.org/jira/browse/DRILL-5126
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Tools, Build & Test
>    Affects Versions: 1.9.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>
> Drill provides a robust selection of test frameworks that have evolved to satisfy the
needs of a variety of test cases. For newbies, however, the result is a bewildering array
of ways to do basically the same thing: set up an embedded Drill cluster, run queries and
check results.
> Further, some key test settings are distributed: some are in the pom.xml file, some in
config files stored as resources, some in hard-coded settings in base test classes.
> Also, some test base classes helpfully set up a test cluster, but then individual tests
need a different config, so they immediately tear down the default cluster and create a new
one.
> This ticket proposes a new test framework, available for new tests, that combines the
best of the existing test frameworks into a single, easy-to-use package.
> * Builder for the cluster
> * Accept config-time options
> * Accept run-time session and system options
> * Specify number of Drillbits
> * Simplified API for the most common options
> * AutoCloseable for use in try-with-resources statements
> * Integration with existing test builder classes
> And so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message