Mailing-List: contact mesos-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mesos-dev@incubator.apache.org
Received-SPF: softfail (nike.apache.org: transitioning domain of
 matei@eecs.berkeley.edu does not designate 169.229.218.145 as permitted
 sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1257)
Subject: Re: how to set up two hdfs for two hadoops
From: Matei Zaharia <matei@eecs.berkeley.edu>
In-Reply-To: 
 <CAPN_WSiGBgU9rZCz1Nebbatf4g4HLopJUi+Msry_5wMd2SV02Q@mail.gmail.com>
Date: Sun, 6 May 2012 20:28:25 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <C2789CD0-DDE7-44E0-AB88-7E4814BBBA95@eecs.berkeley.edu>
References: 
 <CAPN_WSiGBgU9rZCz1Nebbatf4g4HLopJUi+Msry_5wMd2SV02Q@mail.gmail.com>
To: mesos-dev@incubator.apache.org

Hi Nairan,

HDFS doesn't normally run on top of Mesos, and we generally expect =
people to have only one instance of HDFS, which multiple instances of =
MapReduce (or other frameworks) would share. If you want two instances =
of HDFS, you need to set them up manually, and configure them to use =
different ports. Here are the Hadoop settings you need to change:

fs.default.name  (contains port of NameNode)
dfs.http.address  (web UI of NameNode)
dfs.datanode.address
dfs.datanode.ipc.address
dfs.datanode.http.address
dfs.secondary.http.address
dfs.name.dir
dfs.data.dir

We actually do this in our EC2 scripts =
(https://github.com/mesos/mesos/wiki/EC2-Scripts), which will launch a =
Mesos cluster with both a "persistent" and an "ephemeral" HDFS for you. =
You might take a look at how that gets configured.

Matei

On May 6, 2012, at 7:04 PM, Nairan Zhang wrote:

> Hi,
>=20
> It seems it disallows to have the second datanode in one machine. Is =
it a
> common problem? Anybody can please help me out a little bit? Thanks,
>=20
> Nairan