mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "haosdent (JIRA)" <>
Subject [jira] [Commented] (MESOS-3527) HDFS HA fails outside of docker context
Date Sat, 26 Sep 2015 17:32:04 GMT


haosdent commented on MESOS-3527:

What fetcher do here is just call this command to "hadoop fs -copyToLocal" to fetch jars.
So I think your core-site.xml or hdfs-site.xml or environments maybe still incorrect. You
could try "hadoop fs -copyToLocal" in the slave machine shell first to make sure you could
execute it successfully.

> HDFS HA fails outside of docker context
> ---------------------------------------
>                 Key: MESOS-3527
>                 URL:
>             Project: Mesos
>          Issue Type: Bug
>            Reporter: Alan Braithwaite
> I'm using Spark with the Mesos driver.
> When I pass in a `hdfs://<namespace>/path` url in for the spark application, the
fetcher attempts to download the jar files outside the spark context (the docker container
in this case).  The problem is that the core-site.xml and hdfs-site.xml configs exist inside
the container.  The host machine does not have the necessary hdfs configuration to connect
to the HA cluster.
> Currently, I'm not sure what the alternative ways of accessing a HA hadoop cluster besides
through the hadoop client.
> {code}
> I0926 06:34:19.346851 18851 fetcher.cpp:214] Fetching URI 'hdfs://hdfsha/tmp/spark-job.jar'
> I0926 06:34:19.622860 18851 fetcher.cpp:99] Fetching URI 'hdfs://hdfsha/tmp/spark-job.jar'
using Hadoop Client
> I0926 06:34:19.622936 18851 fetcher.cpp:109] Downloading resource from 'hdfs://hdfsha/tmp/spark-job.jar'
to '/state/var/lib/mesos/slaves/20150602-065056-269165578-5050-17724-S12/frameworks/20150914-102037-285942794-5050-31214-0029/executors/driver-20150926063418-0002/runs/9953ae1b-9387-489f-8645-5472d9c5eacf/spark-job.jar'
> E0926 06:34:20.814858 18851 fetcher.cpp:113] HDFS copyToLocal failed: /usr/local/hadoop/bin/hadoop
fs -copyToLocal 'hdfs://hdfsha/tmp/spark-job.jar' '/state/var/lib/mesos/slaves/20150602-065056-269165578-5050-17724-S12/frameworks/20150914-102037-285942794-5050-31214-0029/executors/driver-20150926063418-0002/runs/9953ae1b-9387-489f-8645-5472d9c5eacf/spark-job.jar'
> -copyToLocal: hdfsha
> Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] [-crc] <src>
... <localdst>
> Failed to fetch: hdfs://hdfsha/tmp/spark-job.jar
> {code}
> The code in question:

This message was sent by Atlassian JIRA

View raw message