hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: 0.18.1 datanode psuedo deadlock problem
Date Mon, 12 Jan 2009 20:09:37 GMT
Jason Venner wrote:
> There is no reason to do the block scans. All of the modern kernels will 
> provide you notification when an file or directory is altered.
> 
> This could be readily handled with a native application that writes 
> structured data to a receiver in the Datanode, or via JNA/JNI for pure 
> java or mixed solutions.

Agreed. pure java would be best. I didn't mean to suggest scan as the 
main solution. I only stressed that the problem needs to be fixed.

Unlike DU, since scan would be in our control and it could be throttled 
at our own pace so that it does not thrash the disks. It will have 
minimal effect on DN.

Please open a jira for this issue, so the these solutions could be proposed.

would such a notifier help with thrashing from DU? May be we should have 
alternatives to calling du shell command on the entire directory.

Raghu.

Mime
View raw message