spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Rosen <joshro...@databricks.com>
Subject Re: Spark doesn't unset HADOOP_CONF_DIR when testing ?
Date Sun, 06 Dec 2015 20:32:52 GMT
I agree that we should unset this in our tests. Want to file a JIRA and
submit a PR to do this?

On Thu, Dec 3, 2015 at 6:40 PM Jeff Zhang <zjffdu@gmail.com> wrote:

> I try to do test on HiveSparkSubmitSuite on local box, but fails. The
> cause is that spark is still using my local single node cluster hadoop when
> doing the unit test. I don't think it make sense to do that. These
> environment variable should be unset before the testing. And I suspect
> dev/run-tests also
> didn't do that either.
>
> Here's the error message:
>
> Cause: java.lang.RuntimeException: java.lang.RuntimeException: The root
> scratch dir: /tmp/hive on HDFS should be writable. Current permissions are:
> rwxr-xr-x
> [info]   at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
> [info]   at
> org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:171)
> [info]   at
> org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:162)
> [info]   at
> org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:160)
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Mime
View raw message