hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Does NameNode directory need fine-grained path lock?
Date Tue, 08 May 2012 16:31:59 GMT
Hi Denny,

Do you see an issue in practice?

Fine grained locking makes sense if you hold the lock while doing disk
IO. But since the structure is all in-memory, it's far less important,
and in fact could reduce performance in some cases.

Until you hit a few thousand nodes in your cluster, lock contention in
the NN is not a bottleneck. And the read-write lock introduced in
later versions gets past this issue even for ~4k node clusters for the
most part.

Not to say it won't ever make sense, but seems like a low priority change.


On Tue, May 8, 2012 at 1:15 AM, Denny Ye <dennyy99@gmail.com> wrote:
> hi guys,
>     Currently, NameNode uses read-write lock at top root folder. In my
> opinion, it's too huge for whole namespace with million of file/folder.
> Meanwhile, a few level folder on top of the namespace directory has been set
> with different application. Does we need lesser lock level for such
> application sub-directory to reduce impact each other? It can be used with
> five(or less) top level to HDFS path with individual lock tree-like
> structure.
>      Can anybody provide your feedback or advice? Thanks
> -Regards
> Denny Ye

Todd Lipcon
Software Engineer, Cloudera

View raw message