spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aureliano Buendia <buendia...@gmail.com>
Subject Re: Using persistent hdfs on spark ec2 instanes
Date Thu, 23 Jan 2014 00:36:10 GMT
peristent-hdfs server is set to 9010 port, instead of 9000. Does spark need
more config for this?


On Thu, Jan 23, 2014 at 12:26 AM, Patrick Wendell <pwendell@gmail.com>wrote:

> > 1. It seems by default spark ec2 uses ephemeral hdfs, how to switch this
> to
> > persistent hdfs?
> You can stop the ephemeral one using
>
> /root/ephemeral-hdfs/bin/stop-dfs.sh
>
> and start the persistent one using
>
>  /root/persistent-hdfs/bin/start-dfs.sh
>
> > 2. By default persistent hdfs server is not up, is this meant to be like
> > this?
>
> Yes - it starts only an ephemeral one:
>
> "The spark-ec2 script already sets up a HDFS instance for you. It’s
> installed in /root/ephemeral-hdfs"
>

Mime
View raw message