hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mag Gam <magaw...@gmail.com>
Subject Re: Speeding up Data Deletion From Datanodes
Date Sun, 16 Jan 2011 16:39:15 GMT
I am curious now...

If you have a cluster the size of 10, what should the heartbeat be set
as? What about 100, 1000?


I too am interested in tuning documentation.  For example, how much
memory should we allocate to JVM? How much memory for namenode? etc...



On Thu, Jan 13, 2011 at 1:22 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Hi Sravan,
> You may want to consider backporting HDFS-611 (or using CDH3b3 which
> includes this backport, if you aren't in the mood to patch yourself)
> -Todd
>
> On Thu, Jan 13, 2011 at 9:32 AM, sravankumar <sravankumar@huawei.com> wrote:
>>
>> Hi,
>>
>>
>>
>>             I have gone through the file deletion flow and came to know
>> that
>>
>> Replication Monitor is responsible for File Deletions and these
>> configurations will affect the block deletion
>>
>>
>>
>> INVALIDATE_WORK_PCT_PER_ITERATION
>>
>> BLOCK_INVALIDATE_CHUNK
>>
>>
>>
>>                 Can any one suggest how can we tune up these
>> configurations to speed up block deletion and the significance of
>>
>> INVALIDATE_WORK_PCT_PER_ITERATION constant which by default is 32.
>>
>>
>>
>>                 And also can we tune the heartbeat interval  based
on the
>> cluster size.
>>
>> Suppose it is 10 Node Cluster can some one suggest how can we tune up the
>> configurations. Is there any documentation
>>
>> for the same regarding tuning up of configurations based on the cluster
>> usage.
>>
>>
>>
>> Thanks & Regards,
>>
>> Sravan kumar.
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message