hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <ja...@attributor.com>
Subject Re: 0.18.1 datanode psuedo deadlock problem
Date Mon, 12 Jan 2009 21:00:58 GMT
Here is some simple code I wrote using JNA to handline linux INOTIFY. 
This code was my first and only attempt to use JNA.

The JNA jars are available from  https://jna.dev.java.net/


Raghu Angadi wrote:
> Jason Venner wrote:
>> There is no reason to do the block scans. All of the modern kernels 
>> will provide you notification when an file or directory is altered.
>>
>> This could be readily handled with a native application that writes 
>> structured data to a receiver in the Datanode, or via JNA/JNI for 
>> pure java or mixed solutions.
>
> Agreed. pure java would be best. I didn't mean to suggest scan as the 
> main solution. I only stressed that the problem needs to be fixed.
>
> Unlike DU, since scan would be in our control and it could be 
> throttled at our own pace so that it does not thrash the disks. It 
> will have minimal effect on DN.
>
> Please open a jira for this issue, so the these solutions could be 
> proposed.
>
> would such a notifier help with thrashing from DU? May be we should 
> have alternatives to calling du shell command on the entire directory.
>
> Raghu.

Mime
View raw message