drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5126) Provide simplified, unified "cluster fixture" for tests
Date Sat, 07 Jan 2017 06:27:59 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806928#comment-15806928
] 

ASF GitHub Bot commented on DRILL-5126:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/710#discussion_r95052969
  
    --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/ClusterFixture.java ---
    @@ -0,0 +1,405 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + ******************************************************************************/
    +package org.apache.drill.test;
    +
    +import java.io.File;
    +import java.io.IOException;
    +import java.net.URL;
    +import java.util.ArrayList;
    +import java.util.Collection;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.Map.Entry;
    +import java.util.Properties;
    +
    +import org.apache.commons.io.FileUtils;
    +import org.apache.drill.DrillTestWrapper.TestServices;
    +import org.apache.drill.QueryTestUtil;
    +import org.apache.drill.common.config.DrillConfig;
    +import org.apache.drill.common.exceptions.ExecutionSetupException;
    +import org.apache.drill.exec.ExecConstants;
    +import org.apache.drill.exec.ZookeeperHelper;
    +import org.apache.drill.exec.client.DrillClient;
    +import org.apache.drill.exec.memory.BufferAllocator;
    +import org.apache.drill.exec.memory.RootAllocatorFactory;
    +import org.apache.drill.exec.proto.UserBitShared.QueryType;
    +import org.apache.drill.exec.rpc.user.QueryDataBatch;
    +import org.apache.drill.exec.server.Drillbit;
    +import org.apache.drill.exec.server.RemoteServiceSet;
    +import org.apache.drill.exec.store.StoragePluginRegistry;
    +import org.apache.drill.exec.store.StoragePluginRegistryImpl;
    +import org.apache.drill.exec.store.dfs.FileSystemConfig;
    +import org.apache.drill.exec.store.dfs.FileSystemPlugin;
    +import org.apache.drill.exec.store.dfs.WorkspaceConfig;
    +import org.apache.drill.exec.store.mock.MockStorageEngine;
    +import org.apache.drill.exec.store.mock.MockStorageEngineConfig;
    +import org.apache.drill.exec.util.TestUtilities;
    +
    +import com.google.common.base.Charsets;
    +import com.google.common.base.Preconditions;
    +import com.google.common.io.Files;
    +import com.google.common.io.Resources;
    +
    +/**
    + * Test fixture to start a Drillbit with provide options, create a client,
    + * and execute queries. Can be used in JUnit tests, or in ad-hoc programs.
    + * Provides a builder to set the necessary embedded Drillbit and client
    + * options, then creates the requested Drillbit and client.
    + */
    +
    +public class ClusterFixture implements AutoCloseable {
    +//  private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(ClientFixture.class);
    +  public static final String ENABLE_FULL_CACHE = "drill.exec.test.use-full-cache";
    +  public static final int MAX_WIDTH_PER_NODE = 2;
    +
    +  @SuppressWarnings("serial")
    +  public static final Properties TEST_CONFIGURATIONS = new Properties() {
    +    {
    +      // Properties here mimic those in drill-root/pom.xml, Surefire plugin
    +      // configuration. They allow tests to run successfully in Eclipse.
    +
    +      put(ExecConstants.SYS_STORE_PROVIDER_LOCAL_ENABLE_WRITE, false);
    +      put(ExecConstants.HTTP_ENABLE, false);
    +      put(Drillbit.SYSTEM_OPTIONS_NAME, "org.apache.drill.exec.compile.ClassTransformer.scalar_replacement=on");
    +      put(QueryTestUtil.TEST_QUERY_PRINTING_SILENT, true);
    +      put("drill.catastrophic_to_standard_out", true);
    +
    +      // See Drillbit.close. The Drillbit normally waits a specified amount
    +      // of time for ZK registration to drop. But, embedded Drillbits normally
    +      // don't use ZK, so no need to wait.
    +
    +      put(ExecConstants.ZK_REFRESH, 0);
    +
    +      // This is just a test, no need to be heavy-duty on threads.
    +      // This is the number of server and client RPC threads. The
    +      // production default is DEFAULT_SERVER_RPC_THREADS.
    +
    +      put(ExecConstants.BIT_SERVER_RPC_THREADS, 2);
    +
    +      // No need for many scanners except when explicitly testing that
    +      // behavior. Production default is DEFAULT_SCAN_THREADS
    +
    +      put(ExecConstants.SCAN_THREADPOOL_SIZE, 4);
    +    }
    +  };
    +
    +
    +  public static final String DEFAULT_BIT_NAME = "drillbit";
    +
    +  private DrillConfig config;
    +  private Map<String,Drillbit> bits = new HashMap<>( );
    +  private BufferAllocator allocator;
    +  private boolean ownsZK;
    +  private ZookeeperHelper zkHelper;
    +  private RemoteServiceSet serviceSet;
    +  private String dfsTestTmpSchemaLocation;
    +  List<ClientFixture> clients = new ArrayList<>( );
    +
    +  ClusterFixture( FixtureBuilder  builder ) throws Exception {
    +
    +    // Start ZK if requested.
    +
    +    if (builder.zkHelper != null) {
    +      zkHelper = builder.zkHelper;
    +      ownsZK = false;
    +    } else if (builder.zkCount > 0) {
    +      zkHelper = new ZookeeperHelper(true);
    +      zkHelper.startZookeeper(builder.zkCount);
    +      ownsZK = true;
    +    }
    +
    +    // Create a config
    +    // Because of the way DrillConfig works, we can set the ZK
    +    // connection string only if a property set is provided.
    +
    +    if ( builder.configResource != null ) {
    +      config = DrillConfig.create(builder.configResource);
    +    } else if (builder.configProps != null) {
    +      config = DrillConfig.create(configProperties(builder.configProps));
    +    } else {
    +      config = DrillConfig.create(configProperties(TEST_CONFIGURATIONS));
    +    }
    +
    +    // Not quite sure what this is, but some tests seem to use it.
    +
    +    if (builder.enableFullCache ||
    +        (config.hasPath(ENABLE_FULL_CACHE) && config.getBoolean(ENABLE_FULL_CACHE)))
{
    +      serviceSet = RemoteServiceSet.getServiceSetWithFullCache(config, allocator);
    +    } else {
    +      serviceSet = RemoteServiceSet.getLocalServiceSet();
    +    }
    +
    +    dfsTestTmpSchemaLocation = TestUtilities.createTempDir();
    +
    +    Preconditions.checkArgument(builder.bitCount > 0);
    +    int bitCount = builder.bitCount;
    +    for ( int i = 0;  i < bitCount;  i++ ) {
    +      @SuppressWarnings("resource")
    +      Drillbit bit = new Drillbit(config, serviceSet);
    +      bit.run( );
    +
    +      // Create the dfs_test name space
    +
    +      @SuppressWarnings("resource")
    +      final StoragePluginRegistry pluginRegistry = bit.getContext().getStorage();
    +      TestUtilities.updateDfsTestTmpSchemaLocation(pluginRegistry, dfsTestTmpSchemaLocation);
    +      TestUtilities.makeDfsTmpSchemaImmutable(pluginRegistry);
    +
    +      // Create the mock data plugin
    +      // (Disabled until DRILL-5152 is committed.)
    +
    +//      MockStorageEngineConfig config = MockStorageEngineConfig.INSTANCE;
    +//      @SuppressWarnings("resource")
    +//      MockStorageEngine plugin = new MockStorageEngine(MockStorageEngineConfig.INSTANCE,
bit.getContext(), MockStorageEngineConfig.NAME);
    +//      ((StoragePluginRegistryImpl) pluginRegistry).definePlugin(MockStorageEngineConfig.NAME,
config, plugin);
    +
    +      // Bit name and registration.
    +
    +      String name;
    +      if (builder.bitNames != null && i < builder.bitNames.length) {
    +        name = builder.bitNames[i];
    +      } else {
    +        if (i == 0) {
    +          name = DEFAULT_BIT_NAME;
    +        } else {
    +          name = DEFAULT_BIT_NAME + Integer.toString(i+1);
    +        }
    +      }
    +      bits.put(name, bit);
    +    }
    +
    +    // Some operations need an allocator.
    +
    +    allocator = RootAllocatorFactory.newRoot(config);
    +
    +    // Apply system options
    +    if ( builder.systemOptions != null ) {
    +      for ( FixtureBuilder.RuntimeOption option : builder.systemOptions ) {
    +        clientFixture( ).alterSystem( option.key, option.value );
    +      }
    +    }
    +    // Apply session options.
    +
    +    boolean sawMaxWidth = false;
    +    if ( builder.sessionOptions != null ) {
    +      for ( FixtureBuilder.RuntimeOption option : builder.sessionOptions ) {
    +        clientFixture( ).alterSession( option.key, option.value );
    +        if ( option.key.equals( ExecConstants.MAX_WIDTH_PER_NODE_KEY ) ) {
    +          sawMaxWidth = true;
    +        }
    +      }
    +    }
    +
    +    // Set the default parallelization unless already set by the caller.
    +
    +    if ( ! sawMaxWidth ) {
    +      clientFixture( ).alterSession( ExecConstants.MAX_WIDTH_PER_NODE_KEY, MAX_WIDTH_PER_NODE
);
    +    }
    +  }
    +
    +  private Properties configProperties(Properties configProps) {
    +    Properties effectiveProps = new Properties( );
    +    for (Entry<Object, Object> entry : configProps.entrySet()) {
    +      effectiveProps.put(entry.getKey(), entry.getValue().toString());
    +    }
    +    if (zkHelper != null) {
    +      effectiveProps.put(ExecConstants.ZK_CONNECTION, zkHelper.getConfig().getString(ExecConstants.ZK_CONNECTION));
    +    }
    +    return effectiveProps;
    +  }
    +
    +  public Drillbit drillbit( ) { return bits.get(DEFAULT_BIT_NAME); }
    --- End diff --
    
    Yes, it is a bit messy. Changed this a different way by defining a "default drillbit"
that is returned by this method. That way, we don't rely on the name. (Which, when bits are
named by the caller, we don't know anyway.)


> Provide simplified, unified "cluster fixture" for tests
> -------------------------------------------------------
>
>                 Key: DRILL-5126
>                 URL: https://issues.apache.org/jira/browse/DRILL-5126
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Tools, Build & Test
>    Affects Versions: 1.9.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>
> Drill provides a robust selection of test frameworks that have evolved to satisfy the
needs of a variety of test cases. For newbies, however, the result is a bewildering array
of ways to do basically the same thing: set up an embedded Drill cluster, run queries and
check results.
> Further, some key test settings are distributed: some are in the pom.xml file, some in
config files stored as resources, some in hard-coded settings in base test classes.
> Also, some test base classes helpfully set up a test cluster, but then individual tests
need a different config, so they immediately tear down the default cluster and create a new
one.
> This ticket proposes a new test framework, available for new tests, that combines the
best of the existing test frameworks into a single, easy-to-use package.
> * Builder for the cluster
> * Accept config-time options
> * Accept run-time session and system options
> * Specify number of Drillbits
> * Simplified API for the most common options
> * AutoCloseable for use in try-with-resources statements
> * Integration with existing test builder classes
> And so on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message