hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <>
Subject [jira] [Created] (HIVE-17111) TestSparkCliDriver does not use LocalHiveSparkClient
Date Mon, 17 Jul 2017 20:14:00 GMT
Sahil Takiar created HIVE-17111:

             Summary: TestSparkCliDriver does not use LocalHiveSparkClient
                 Key: HIVE-17111
             Project: Hive
          Issue Type: Bug
          Components: Spark
            Reporter: Sahil Takiar
            Assignee: Sahil Takiar

The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but the HoS still
uses decides to use the RemoteHiveSparkClient rather than the LocalHiveSparkClient.

The issue is with the following check in HiveSparkClientFactory:

    if (master.equals("local") || master.startsWith("local[")) {
      // With local spark context, all user sessions share the same spark context.
      return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
    } else {
      return new RemoteHiveSparkClient(hiveconf, sparkConf);

When {{master.startsWith("local[")}} it checks the value of spark.master and sees that it
doesn't start with {{local[}} and then decides to use the RemoteHiveSparkClient.

We should fix this so that the LocalHiveSparkClient is used. It should speed up some of the
tests, and also makes qtests easier to debug since everything will now be run in the same

This message was sent by Atlassian JIRA

View raw message