spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <>
Subject incorrect message that path appears to be local
Date Wed, 12 Oct 2016 18:55:53 GMT
i see this warning when running jobs on cluster:

2016-10-12 14:46:47 WARN spark.SparkContext: Spark is not running in local
mode, therefore the checkpoint directory must not be on the local
filesystem. Directory '/tmp' appears to be on the local filesystem.

however the checkpoint "directory" that it warns about is a hadoop path. i
use an unqualified path, which means a path on the default filesystem by
hadoop convention. when running on the cluster my default filesystem is
hdfs (and it correctly uses hdfs).

how about if we change the method that does this check
(Utils.nonLocalPaths) to be aware of the default filesystem instead of
incorrectly assuming its local if not specified?

View raw message