phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3181) Run test methods in parallel to reduce test suite run time
Date Tue, 16 Aug 2016 22:26:20 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423509#comment-15423509
] 

James Taylor commented on PHOENIX-3181:
---------------------------------------

In order to get reasonable parallelization from the mini cluster that is spun up, you'll need
to *not* override some of the config properties we set (as we set them on purpose to lower
values as we're not currently expecting to need any parallelization). For example
- In BaseTest.setUpConfigForMiniCluster, remove the code adjusting the config properties down:
{code}
    public static Configuration setUpConfigForMiniCluster(Configuration conf, ReadOnlyProps
overrideProps) {
        assertNotNull(conf);
        setDefaultTestConfig(conf, overrideProps);
        /*
         * The default configuration of mini cluster ends up spawning a lot of threads
         * that are not really needed by phoenix for test purposes. Limiting these threads
         * helps us in running several mini clusters at the same time without hitting 
         * the threads limit imposed by the OS. 
         */
        conf.setInt(HConstants.REGION_SERVER_HANDLER_COUNT, 5);
        conf.setInt("hbase.regionserver.metahandler.count", 2);
        conf.setInt(HConstants.MASTER_HANDLER_COUNT, 2);
        conf.setInt("dfs.namenode.handler.count", 2);
        conf.setInt("dfs.namenode.service.handler.count", 2);
        conf.setInt("dfs.datanode.handler.count", 2);
        conf.setInt("ipc.server.read.threadpool.size", 2);
        conf.setInt("ipc.server.handler.threadpool.size", 2);
        conf.setInt("hbase.regionserver.hlog.syncer.count", 2);
        conf.setInt("hbase.hlog.asyncer.number", 2);
        conf.setInt("hbase.assignment.zkevent.workers", 5);
        conf.setInt("hbase.assignment.threads.max", 5);
        conf.setInt("hbase.catalogjanitor.interval", 5000);
        return conf;
    }
{code}
- Remove the overriding/setting of these properties in QueryServicesTestImpl (so that the
regular non test defaults are used):
{code}
    private static final int DEFAULT_THREAD_POOL_SIZE = 20;
    private static final int DEFAULT_QUEUE_SIZE = 0;
    ....
    public static final int DEFAULT_INDEX_HANDLER_COUNT = 5;
    public static final int DEFAULT_METADATA_HANDLER_COUNT = 5;
    public static final int DEFAULT_HCONNECTION_POOL_CORE_SIZE = 10;
    public static final int DEFAULT_HCONNECTION_POOL_MAX_SIZE = 10;
    public static final int DEFAULT_HTABLE_MAX_THREADS = 10;
{code}

> Run test methods in parallel to reduce test suite run time
> ----------------------------------------------------------
>
>                 Key: PHOENIX-3181
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3181
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>
> With PHOENIX-2995, the tests within a test class can be executed in parallel since the
table names won't conflict. This should greatly reduce the time taken to run all of our tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message