hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Kleinpaste <k...@conviva.com>
Subject Re: problem with completion notification from block movement
Date Mon, 02 Feb 2009 18:23:24 GMT
On Sun, 2009-02-01 at 17:58 -0800, jason hadoop wrote:
> The Datanode's use multiple threads with locking and one of the
> assumptions is that the block report (1ce per hour by default) takes
> little time. The datanode will pause while the block report is running
> and if it happens to take a while weird things start to happen.

Thank you for responding, this is very informative for us.

Having looked through the source code with a co-worker regarding
periodic scan and then checking the logs once again, we find that we are
finding reports of this sort:

BlockReport of 1158499 blocks got processed in 308860 msecs
BlockReport of 1159840 blocks got processed in 237925 msecs
BlockReport of 1161274 blocks got processed in 177853 msecs
BlockReport of 1162408 blocks got processed in 285094 msecs
BlockReport of 1164194 blocks got processed in 184478 msecs
BlockReport of 1165673 blocks got processed in 226401 msecs

The 3rd of these exactly straddles the particular example timeline I
discussed in my original email about this question.  I suspect I'll find
more of the same as I look through other related errors.


View raw message