Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of mirko.kaempf@gmail.com
 designates 209.85.192.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <g2TY5kS6.1427296268.7782680.peridale@localhost>
References: <g2TY5kS6.1427296268.7782680.peridale@localhost>
From: =?UTF-8?B?TWlya28gS8OkbXBm?= <mirko.kaempf@gmail.com>
Date: Wed, 25 Mar 2015 15:20:03 +0000
Message-ID: 
 <CALKHTG9tfPW4j1fniSofVj+=Bnzvr0KhHFUPgGOfxOLqcrpBsA@mail.gmail.com>
Subject: Re: can block size for namenode be different from datanode block
 size?
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=047d7bfeac5226480b05121e70bb

--047d7bfeac5226480b05121e70bb
Content-Type: text/plain; charset=UTF-8

Hi Mich,

please see the comments in your text.


2015-03-25 15:11 GMT+00:00 Dr Mich Talebzadeh <mich@peridale.co.uk>:

>
> Hi,
>
> The block size for HDFS is currently set to 128MB by defauilt. This is
> configurable.
>
Correct, an HDFS client can overwrite the cfg-property and define a
different block size for HDFS blocks.

>
> My point is that I assume this  parameter in hadoop-core.xml sets the
> block size for both namenode and datanode.

Correct, the block-size is a "HDFS wide setting" but in general the
HDFS-client makes the blocks.


> However, the storage and
> random access for metadata in nsamenode is different and suits smaller
> block sizes.
>
HDFS blocksize has no impact here. NameNode metadata is held in memory. For
reliability it is dumped to local discs of the server.


>
> For example in Linux the OS block size is 4k which means one HTFS blopck
> size  of 128MB can hold 32K OS blocks. For metadata this may not be
> useful and smaller block size will be suitable and hence my question.
>
Remember, metadata is in memory. The fsimage-file, which contains the
metadata
is loaded on startup of the NameNode.

Please be not confused by the two types of block-sizes.

Hope this helps a bit.
Cheers,
Mirko


>
> Thanks,
>
> Mich
>

--047d7bfeac5226480b05121e70bb
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Mich,<br><div class=3D"gmail_extra"><br></div><div clas=
s=3D"gmail_extra">please see the comments in your text.<br><br></div><div c=
lass=3D"gmail_extra"><br></div><div class=3D"gmail_extra">=C2=A0</div><div =
class=3D"gmail_extra"><div class=3D"gmail_quote">2015-03-25 15:11 GMT+00:00=
 Dr Mich Talebzadeh <span dir=3D"ltr">&lt;<a href=3D"mailto:mich@peridale.c=
o.uk" target=3D"_blank">mich@peridale.co.uk</a>&gt;</span>:<br><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid=
;padding-left:1ex"><br>
Hi,<br>
<br>
The block size for HDFS is currently set to 128MB by defauilt. This is<br>
configurable.<br></blockquote><div>Correct, an HDFS client can overwrite th=
e cfg-property and define a different block size for HDFS blocks.=C2=A0</di=
v><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex">
<br>
My point is that I assume this=C2=A0 parameter in hadoop-core.xml sets the<=
br>
block size for both namenode and datanode. </blockquote><div>Correct, the b=
lock-size is a &quot;HDFS wide setting&quot; but in general the HDFS-client=
 makes the blocks.<br>=C2=A0=C2=A0</div><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Howe=
ver, the storage and<br>
random access for metadata in nsamenode is different and suits smaller<br>
block sizes.<br></blockquote><div>HDFS blocksize has no impact here. NameNo=
de metadata is held in memory. For reliability it is dumped to local discs =
of the server.<br>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"ma=
rgin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
For example in Linux the OS block size is 4k which means one HTFS blopck<br=
>
size=C2=A0 of 128MB can hold 32K OS blocks. For metadata this may not be<br=
>
useful and smaller block size will be suitable and hence my question.<br></=
blockquote><div>Remember, metadata is in memory. The fsimage-file, which co=
ntains the metadata <br>is loaded on startup of the NameNode.</div><div><br=
></div><div>Please be not confused by the two types of block-sizes.</div><d=
iv><br></div><div>Hope this helps a bit.</div><div>Cheers,</div><div>Mirko<=
/div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0=
 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks,<br>
<br>
Mich<br>
</blockquote></div><br></div></div>

--047d7bfeac5226480b05121e70bb--