hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Subramaniyan <hsubramani...@hortonworks.com>
Subject Re: problems running spark tests
Date Thu, 02 Jul 2015 18:54:35 GMT
Can you try running with -Dspark.query.files instead of -Dqfile from itests directory

Thanks
Hari

> On Jul 2, 2015, at 10:32 AM, "Sergey Shelukhin" <sergey@hortonworks.com> wrote:
> 
> Hi. I am trying to run TestSparkCliDriver.
> 
> 1) Spark tests do not appear to support specifying a query like other
> tests; when I run mvn test -Phadoop-2 -Dtest=TestSparkCliDriver tests run,
> but with 
> mvn test -Phadoop-2 -Dtest=TestSparkCliDriver -Dqfile=foo.q,bar.q,.. test
> just instantly succeeds w/o running any queries. Is there some other way
> to specify those?
> 
> 2) When I run all the test, they fail with the below exception
> I’ve done a full regular build (mvn clean install … in root and then
> itests). Are more steps necessary?
> The itests/qtest-spark/../../itests/qtest-spark/target/spark directory
> exists and has bunch of stuff, but bin/ subdirectory that it tries to run
> from is indeed empty.
> 
> 2015-07-02 10:11:58,678 ERROR [main]: spark.SparkTask
> (SessionState.java:printError(987)) - Failed to execute spark task, with
> exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to
> create spark client.)'
> org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark
> client.
> at 
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSes
> sionImpl.java:57)
> at 
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSes
> sion(SparkSessionManagerImpl.java:114)
> at 
> org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUt
> ilities.java:127)
> at 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:101)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1672)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1431)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1212)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1063)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1053)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:840)
> at 
> org.apache.hadoop.hive.cli.TestSparkCliDriver.<clinit>(TestSparkCliDriver.j
> ava:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:6
> 2)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
> l.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java
> :35)
> at org.junit.internal.runners.SuiteMethod.<init>(SuiteMethod.java:24)
> at 
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBu
> ilder.java:11)
> at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java
> :59)
> at 
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(A
> llDefaultPossibilitiesBuilder.java:26)
> at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java
> :59)
> at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java
> :262)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provid
> er.java:153)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:
> 124)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoad
> er(ForkedBooter.java:200)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBoot
> er.java:153)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.io.IOException: Cannot run program
> “[snip]/itests/qtest-spark/../../itests/qtest-spark/target/spark/bin/spark-
> submit": error=2, No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at 
> org.apache.hive.spark.client.SparkClientImpl.startDriver(SparkClientImpl.ja
> va:415)
> at 
> org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:94
> )
> at 
> org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFac
> tory.java:80)
> at 
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiv
> eSparkClient.java:91)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSpark
> Client(HiveSparkClientFactory.java:65)
> at 
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSes
> sionImpl.java:55)
> ... 33 more
> Caused by: java.io.IOException: error=2, No such file or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 39 more
> 
> 
> 

Mime
View raw message