hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: Big HDFS deletes lead to dead datanodes
Date Tue, 24 Feb 2009 19:40:07 GMT

What is the Hadoop version? DN limits deletes per heartbeat to 100 or so 
I think. So the dead datanodes might not be dieing only because of 
deletes... does stacktrace show that?

 > [...] Ideally we would never see "dead" datanodes from doing deletes.

yes : HADOOP-4584 moves deletions out of heartbeat thread.


Bryan Duxbury wrote:
> On occasion, I've deleted a few TB of stuff in DFS at once. I've noticed 
> that when I do this, datanodes start taking a really long time to check 
> in and ultimately get marked dead. Some time later, they'll get done 
> deleting stuff and come back and get unmarked.
> I'm wondering, why do deletions get more priority than checking in? 

> -Bryan

View raw message