hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho (JIRA)" <>
Subject [jira] [Commented] (HIVE-7271) Speed up unit tests
Date Sun, 22 Jun 2014 21:36:24 GMT


Szehon Ho commented on HIVE-7271:

Today the ptest host instance is size C1.xlarge (7gb).

Glad somebody is looking at it, also had minor comments.

1.  We shouldn't remove the hadoopShims check of local mode, in MapRedTask right?  Shouldn't
user still able to set this via the mr-specific flags?

1.  Not related to this change, but can this method name be refactored as it's confusing (now
its executing in process and not in child jvm)
+    if (!conf.getBoolVar(HiveConf.ConfVars.SUBMITLOCALTASKVIACHILD)) {
+      // execute in process
+      return executeFromChildJVM(driverContext);
+    }

> Speed up unit tests
> -------------------
>                 Key: HIVE-7271
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Gunther Hagleitner
>            Assignee: Gunther Hagleitner
>         Attachments: HIVE-7271.1.patch
> Did some experiments to see if there's a way to speed up unit tests. TestCliDriver seemed
to take a lot of time just spinning up/tearing down JVMs. I was also curious to see if running
everything on a ram disk would help.
> Results (I ran tests up to authorization_2):
> - Current setup: 40 minutes
> - Single JVM (not using child JVM to run all queries): 8 minutes
> - Single JVM + ram disk: 7 minutes
> So the ram disk didn't help that much. But running tests in single JVM seems worthwhile

This message was sent by Atlassian JIRA

View raw message