hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <ja...@attributor.com>
Subject Re: 0.18.1 datanode psuedo deadlock problem
Date Mon, 12 Jan 2009 20:47:38 GMT
The thought is that the notifier would stat each file as it was notified 
about it, and thus would have the real time dusage information also.
There would be no need for the current du task or the block task after 
startup (ie: do it one time to compute the current blocks and space).

After startup the recipient of the notifications would keep up the block 
information and the du information.

Raghu Angadi wrote:
> Jason Venner wrote:
>> There is no reason to do the block scans. All of the modern kernels 
>> will provide you notification when an file or directory is altered.
>>
>> This could be readily handled with a native application that writes 
>> structured data to a receiver in the Datanode, or via JNA/JNI for 
>> pure java or mixed solutions.
>
> Agreed. pure java would be best. I didn't mean to suggest scan as the 
> main solution. I only stressed that the problem needs to be fixed.
>
> Unlike DU, since scan would be in our control and it could be 
> throttled at our own pace so that it does not thrash the disks. It 
> will have minimal effect on DN.
>
> Please open a jira for this issue, so the these solutions could be 
> proposed.
>
> would such a notifier help with thrashing from DU? May be we should 
> have alternatives to calling du shell command on the entire directory.
>
> Raghu.

Mime
View raw message