hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric <eric.x...@gmail.com>
Subject Re: Multiple directories for hadoop
Date Tue, 18 Jan 2011 15:49:24 GMT
Hadoop knows that these are seperate disks (don't ask me how).
BTW,I would mount under something like /mnt/hadoop/disk[0...n]
In the latest Hadoop release from Cloudera for example, they say this:

"In CDH3 Beta 3, the mapred.system.dir directory must be located inside a
directory that is owned by mapred. For
example, if mapred.system.dir is specified as /mapred/system, then /mapred
must be owned by mapred. Don't,
for example, specify /mrsystem as mapred.system.dir because you don't want /
owned by mapred."


2011/1/18 shixing <paradisehit@gmail.com>

> you configue is right, such like:
> <property>
>  <name>dfs.data.dir</name>
>  <value>/data1,/data2,/data3,/data4</value>
> </property>
>
> So the hdfs's block data can stored in the dirs above.
>
> If you want local jobcache, you can use DistributedCache to archive the
> data
> as cache.
> See detail
>
> http://hadoop.apache.org/mapreduce/docs/r0.21.0/mapred_tutorial.html#DistributedCache
>
> By the way, this is a hadoop question, not hbase :)
>
> On Mon, Jan 17, 2011 at 11:01 PM, rajgopalv <raja.fire@gmail.com> wrote:
>
> >
> > I have a doubt in configuring dfs.data.dir,
> >
> > One of my slave has 4 500GB harddisks. They are mounted on different
> mount
> > points : /data1 /data2 /data3 /data4
> > How can i make use of all the 4 harddisks for hdfs data and local jobcahe
> ?
> >
> > if i give comma seperated values for dfs.data.dir, will the total data be
> > replicated on all the 4 disks? , or the total data will be shared on the
> 4
> > disks (without replication) ?
> >
> > And, how to increase the space for local jobcache?
> > http://www.mail-archive.com/core-user@hadoop.apache.org/msg04346.html
> > says hadoop.tmp.dir cannot be comma seperated.  But my mapreduce jobs
> will
> > eat a lot of jobcache in the local.
> >
> > How should the configurartion be for the above scenario?
> >
> > --
> > View this message in context:
> >
> http://old.nabble.com/Multiple-directories-for-hadoop-tp30676207p30676207.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
> >
>
>
> --
> Best wishes!
> My Friend~
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message