hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9900) LLAP: Integrate MiniLLAPCluster into tests
Date Tue, 01 Sep 2015 20:59:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726178#comment-14726178
] 

Siddharth Seth commented on HIVE-9900:
--------------------------------------

Posting comments here since ReviewBoard diff is currently broken.

{code}
80% may be too much, especially for smaller JVM heap sizes
{code}
80% may be a little high, given 15% is taken up by the cache. 70/75 would be a more reliable
value. This really depends on the overall JVM heap size though.

llapCluster.serviceStop(); - should be llapCluster.stop() (That takes care of shutting down
additional services which are part of the LlapDaemon composite service)

{code}
+    llapDaemon = new LlapDaemon(conf, numExecutorsPerService, execBytesPerService, llapIoEnabled,
+        ioIsDirect, ioBytesPerService, localDirs, LlapConfiguration.LLAP_DAEMON_RPC_PORT_DEFAULT,
+        LlapConfiguration.LLAP_DAEMON_YARN_SHUFFLE_PORT_DEFAULT);
{code}
The ports should be left at 0, to set them up dynamically. Calling getClusterSpecificConfiguration
and merging it in with the main conf will provide the randomly selected ports. Otherwise,
multiple instances of the test cannot run on the same node and causes problems with jenkins
and other systems which may be using the pre-defined default ports.

May want to set the queue size to a value below the default of 10.

Orthogonal to some extent: Is it possible to run the tests on MiniLlap with two different
configurations, one with a single executor and a small queue size. This has worked very well
in the past to detect deadlocks and scheduling issues.
Maybe this could be a command line parameter for the test - which can be run on demand when
required.


> LLAP: Integrate MiniLLAPCluster into tests
> ------------------------------------------
>
>                 Key: HIVE-9900
>                 URL: https://issues.apache.org/jira/browse/HIVE-9900
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Siddharth Seth
>            Assignee: Prasanth Jayachandran
>             Fix For: llap
>
>         Attachments: HIVE-9900.1.patch, HIVE-MiniLlapCluster.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message