hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yunjiong zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
Date Wed, 01 Mar 2017 22:28:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891198#comment-15891198
] 

yunjiong zhao commented on HDFS-11384:
--------------------------------------

Thank you [~benoyantony] for your time to review this patch.
{quote}Sleeping inside the Synchronized block should be avoided as it will prevent other threads
from obtaining the lock while the thread is sleeping. {quote}
I did it on purpose for sleeping inside the Synchronized block.
In balancer there are multiple threads (by default 200) that may call getBlocks at same time,
if user need to set dfs.balancer.getBlocks.interval.millis to slow down balancer, without
a lock it won't work well due to at worst case there are still 200 getBlocks send to NameNode
at same time.

{quote}It will be better to keep track of the interval between successive getBlocks and sleep
only for the required time. {quote}
Since by default, this patch doesn't change anything, only add a option let user slow down
balancer send getBlocks to NameNode, so I'd like to keep it as simple as possible.

> Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength
spike
> -------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-11384
>                 URL: https://issues.apache.org/jira/browse/HDFS-11384
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover
>    Affects Versions: 2.7.3
>            Reporter: yunjiong zhao
>            Assignee: yunjiong zhao
>         Attachments: balancer.day.png, balancer.week.png, HDFS-11384.001.patch
>
>
> When running balancer on hadoop cluster which have more than 3000 Datanodes will cause
NameNode's rpc.CallQueueLength spike. We observed this situation could cause Hbase cluster
failure due to RegionServer's WAL timeout.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message