drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5260) Refinements to new "Cluster Fixture" test framework
Date Thu, 23 Feb 2017 21:08:44 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881267#comment-15881267
] 

ASF GitHub Bot commented on DRILL-5260:
---------------------------------------

Github user sohami commented on a diff in the pull request:

    https://github.com/apache/drill/pull/753#discussion_r102587851
  
    --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/ClusterFixture.java ---
    @@ -99,33 +111,83 @@
           // behavior. Production default is DEFAULT_SCAN_THREADS
     
           put(ExecConstants.SCAN_THREADPOOL_SIZE, 4);
    +
    +      // Define a useful root location for the ZK persistent
    +      // storage. Profiles will go here when running in distributed
    +      // mode.
    +
    +      put(ZookeeperPersistentStoreProvider.DRILL_EXEC_SYS_STORE_PROVIDER_ZK_BLOBROOT,
"/tmp/drill/log");
         }
       };
     
       public static final String DEFAULT_BIT_NAME = "drillbit";
     
       private DrillConfig config;
    -  private Map<String,Drillbit> bits = new HashMap<>();
    +  private Map<String, Drillbit> bits = new HashMap<>();
       private Drillbit defaultDrillbit;
       private BufferAllocator allocator;
       private boolean ownsZK;
       private ZookeeperHelper zkHelper;
       private RemoteServiceSet serviceSet;
    -  private String dfsTestTmpSchemaLocation;
    +  private File dfsTestTempDir;
       protected List<ClientFixture> clients = new ArrayList<>();
    +  private boolean usesZk;
    +  private boolean preserveLocalFiles;
    +  private boolean isLocal;
    +
    +  /**
    +   * Temporary directories created for this test cluster.
    +   * Each is removed when closing the cluster.
    +   */
    +
    +  private List<File> tempDirs = new ArrayList<>();
    +
    +  ClusterFixture(FixtureBuilder builder) {
    +
    +    String zkConnect = configureZk(builder);
    +    try {
    +      createConfig(builder, zkConnect);
    +      startDrillbits(builder);
    +      applyOptions(builder);
    +    } catch (Exception e) {
    +      // Translate exceptions to unchecked to avoid cluttering
    +      // tests. Failures will simply fail the test itself.
    +
    +      throw new IllegalStateException( "Cluster fixture setup failed", e );
    +    }
    +
    +    // Some operations need an allocator.
     
    -  protected ClusterFixture(FixtureBuilder  builder) throws Exception {
    +    allocator = RootAllocatorFactory.newRoot(config);
    +  }
    +
    +  private String configureZk(FixtureBuilder builder) {
     
         // Start ZK if requested.
     
    +    String zkConnect = null;
         if (builder.zkHelper != null) {
    +      // Case where the test itself started ZK and we're only using it.
    +
           zkHelper = builder.zkHelper;
           ownsZK = false;
    -    } else if (builder.zkCount > 0) {
    -      zkHelper = new ZookeeperHelper(true);
    -      zkHelper.startZookeeper(builder.zkCount);
    +    } else if (builder.localZkCount > 0) {
    +      // Case where we need a local ZK just for this test cluster.
    +
    +      zkHelper = new ZookeeperHelper("dummy");
    +      zkHelper.startZookeeper(builder.localZkCount);
           ownsZK = true;
         }
    +    if (zkHelper != null) {
    +      zkConnect = zkHelper.getConnectionString();
    +      // Forced to disable this, because currently we leak memory which is a known issue
for query cancellations.
    +      // Setting this causes unittests to fail.
    +      builder.configProperty(ExecConstants.RETURN_ERROR_FOR_FAILURE_IN_CANCELLED_FRAGMENTS,
true);
    +    }
    +    return zkConnect;
    --- End diff --
    
    We can set the `ZK_CONNECTION` property inside configProperty here itself. That way we
don't have to get the connectString from here and pass to createConfig where we are doing
`null` check for `zkConnect` string.


> Refinements to new "Cluster Fixture" test framework
> ---------------------------------------------------
>
>                 Key: DRILL-5260
>                 URL: https://issues.apache.org/jira/browse/DRILL-5260
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.10
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>             Fix For: 1.10
>
>
> Roll-up of a number of enhancements to the cluster fixture framework.
> * Config option to suppress printing of CSV and other output. (Allows printing for single
tests, not printing when running from Maven.)
> * Parsing of query profiles to extract plan and run time information.
> * Fix bug in log fixture when enabling logging for a package.
> * Improved ZK support.
> * Set up the new CTTAS default temporary workspace for tests.
> * Revise TestDrillbitResiliance to use the new framework.
> * Revise TestWindowFrame to to use the new framework.
> * Revise TestMergeJoinWithSchemaChanges to use the new framework.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message