hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Loddengaard <a...@cloudera.com>
Subject Re: Question about Data Node configuration
Date Wed, 07 Jul 2010 18:22:04 GMT
I would recommend not putting / in dfs.data.dir.  You'll want that space for
logs, which will grow very large in heavily-used clusters (userlogs in
particular).

/ for OS and logs
/mount* for mapred.local.dir and dfs.data.dir

Hope this helps.

Alex

On Wed, Jul 7, 2010 at 10:38 AM, A Levine <levine.a@gmail.com> wrote:

> I am trying to configure a large install and I have a question about
> the configuration of Data Nodes.  Each data node has multiple drives.
> Each drive is 1TB in size.  In the hdfs-site.xml, I can have multiple
> directories (which will be mounted drives) specified as shown by:
>
>  <property>
>    <name>dfs.data.dir</name>
>    <value>/mount1,/mount2,/mount3,....</value>
>    <final>true</final>
>  </property>
>
> For the drive that has the OS, only 100G will be used for the OS.  Is
> it good practice to have a partition on the drive that has the OS used
> for the dfs.data.dir?  Will this slow things down?  Will the size
> difference available to each directory be a problem?  Also, if it is
> not a good idea to use the OS drive, then how about pointing logs to
> that drive?
>
> andrew
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message