hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaojinchao <gaojinc...@huawei.com>
Subject Re: A question about dfs.replication.min setting.
Date Sat, 26 Nov 2011 08:14:18 GMT
Thanks for your response. 
My case is complicated. It is not the issue of xceivers.
The main reason of asynchronously I/O, When we write some data to an "async" mounted partition(that
is default in os), such as "create", "mkdir", "rmdir", "rename" and "write". These operations
need a period time to sync. If one machine crashes, will may have some files or some blocks
lose. If setting "sync" affects all I/O to the file system (both meta data and file contents),
performance is low. 
I agree with Harsh, If sets "dfs.replication.min" to 2. It may cause RSes to go down. So I
want to know how set the " dfs.replication.min" in production. Who has experience, Whether
by increasing the value to increase reliability?

Cheers.


-----邮件原件-----
发件人: Jahangir Mohammed [mailto:md.jahangir27@gmail.com] 
发送时间: 2011年11月26日 15:22
收件人: user@hbase.apache.org
主题: Re: A question about dfs.replication.min setting.

Since you say you see blocks loosing when it's IO busy/bound, I very much
that think the xceivers has been set to low value. Raise its value.

Thanks,
Jahangir Mohammed.

On Sat, Nov 26, 2011 at 2:21 AM, Jahangir Mohammed
<md.jahangir27@gmail.com>wrote:

> What is  dfs.datanode.max.xceivers set?
>
>
>
> On Sat, Nov 26, 2011 at 2:17 AM, Harsh J <harsh@cloudera.com> wrote:
>
>> Ah wait, my bad. Do not raise dfs.replication.min when using HBase - it
>> can cause RSes to go down if min block #s > 1 were not completely
>> guaranteed. As a result, close() on files fail to work and block out until
>> the replicas are available to satisfy dfs.replication.min - and thereby
>> cause things to timeout/fail.
>>
>> (Think the problem is to do with use of sync, but am not sure yet --
>> general writes work properly with that config, by retrying enough times to
>> get locations).
>>
>> On 26-Nov-2011, at 12:14 PM, Gaojinchao wrote:
>>
>> > When HBase use HDFS system file. How do we set "dfs.replication.min"?
>> > who can share relevant experience?
>> > Currently on our environment, We use the default value:
>> > dfs.replication :3
>> > dfs.replication.min: 1
>> > I found some block lost when the IO is very busy.
>> >
>> >
>> >
>> >
>>
>>
>
Mime
View raw message