hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <>
Subject [jira] [Commented] (HIVE-17111) TestSparkCliDriver does not use LocalHiveSparkClient
Date Tue, 18 Jul 2017 03:15:00 GMT


Sahil Takiar commented on HIVE-17111:

I'm good either way. While TestMiniSparkOnYarnCliDriver is closer to what is used in production,
a lot of qtests probably don't need to be run on a mini-cluster of processes. If everything
is run in the same process, the test coverage is probably similar. I think Hive-on-MR qtests
is all in process, and some of the LLAP qtests are in process too. The main advantage is reduced
time to run the tests.

That being said, I'm good with either approach. Basically, I think there should just be a
way to run a HoS qtest entirely in process (could just be a config option). It can help a
lot with debugging.

> TestSparkCliDriver does not use LocalHiveSparkClient
> ----------------------------------------------------
>                 Key: HIVE-17111
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but the HoS still
uses decides to use the RemoteHiveSparkClient rather than the LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
>     if (master.equals("local") || master.startsWith("local[")) {
>       // With local spark context, all user sessions share the same spark context.
>       return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
>     } else {
>       return new RemoteHiveSparkClient(hiveconf, sparkConf);
>     }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and sees that
it doesn't start with {{local[}} and then decides to use the RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed up some
of the tests, and also makes qtests easier to debug since everything will now be run in the
same process.

This message was sent by Atlassian JIRA

View raw message