Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Koert Kuipers <koert@tresata.com>
Date: Wed, 12 Oct 2016 14:55:53 -0400
Message-ID: <CANx3uAhxwYbhaHxipzS-0YP00P_ydxZrRqJBotvpR2UkBBO4bg@mail.gmail.com>
Subject: incorrect message that path appears to be local
To: "dev@spark.apache.org" <dev@spark.apache.org>
Content-Type: multipart/alternative; boundary=e89a8f6432b2e54b5f053eaf8adf
archived-at: Wed, 12 Oct 2016 18:56:01 -0000

--e89a8f6432b2e54b5f053eaf8adf
Content-Type: text/plain; charset=UTF-8

i see this warning when running jobs on cluster:

2016-10-12 14:46:47 WARN spark.SparkContext: Spark is not running in local
mode, therefore the checkpoint directory must not be on the local
filesystem. Directory '/tmp' appears to be on the local filesystem.

however the checkpoint "directory" that it warns about is a hadoop path. i
use an unqualified path, which means a path on the default filesystem by
hadoop convention. when running on the cluster my default filesystem is
hdfs (and it correctly uses hdfs).

how about if we change the method that does this check
(Utils.nonLocalPaths) to be aware of the default filesystem instead of
incorrectly assuming its local if not specified?

--e89a8f6432b2e54b5f053eaf8adf
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:courier =
new,monospace">i see this warning when running jobs on cluster:<br><br>2016=
-10-12 14:46:47 WARN spark.SparkContext: Spark is not running in local mode=
, therefore the checkpoint directory must not be on the local filesystem. D=
irectory &#39;/tmp&#39; appears to be on the local filesystem.<br><br></div=
><div class=3D"gmail_default" style=3D"font-family:courier new,monospace">h=
owever the checkpoint &quot;directory&quot; that it warns about is a hadoop=
 path. i use an unqualified path, which means a path on the default filesys=
tem by hadoop convention. when running on the cluster my default filesystem=
 is hdfs (and it correctly uses hdfs).<br><br></div><div class=3D"gmail_def=
ault" style=3D"font-family:courier new,monospace">how about if we change th=
e method that does this check (Utils.nonLocalPaths) to be aware of the defa=
ult filesystem instead of incorrectly assuming its local if not specified?<=
br><br></div></div>

--e89a8f6432b2e54b5f053eaf8adf--